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1 METHOD AND APPARATUS FOR PLACING VIRTUAL OBJECTS 

2 This application is a continuation-in-part of U.S. Application Serial Number 

3 09/597,893, filed June 19, 2000, entitled METHOD AND APPARATUS FOR 

4 TARGETING VIRTUAL OBJECTS, which is a continuation-in-part of U.S. Application 

5 Serial Number 09/054,4 19, filed April 3, 1998, entitled TARGETED ADVERTISEMENT 

6 USING TELEVISION DELIVERY SYSTEMS, which is a continuation-in-part of U.S. 

7 Application Serial Number 08/735,549, filed October 23, 1996, entitled METHOD AND 

8 APPARATUS FOR TARGETED ADVERTISING, which is a continuation of U.S. 

9 Application Serial Number 08/160,280, filed December 2, 1993, entitled NETWORK 

10 CONTROLLER FOR CABLE TELEVISION DELIVERY SYSTEM, now U.S. Patent 

1 1 Number 5,600,364, which was a continuation-in-part of U.S. Application Serial Number 

1 2 07/99 1 ,074, filed December 9, 1 992, entitled REMOTE CONTROL FOR MENU DRIVEN 

1 3 SUBSCRIBER ACCESS TO TELEVISION PROGRAMMING. These patents and patent 

14 applications are hereby incorporated by reference. 

15 Technical Field 

1 6 The technical field relates to a method and apparatus for providing virtual objects for 

1 7 display at a terminal. The method and apparatus specifically relate to monitoring, controlling 

1 8 and managing a content delivery network including an operations center, a local insertion 

19 center, or a subscriber's local terminal. 

20 Background 

2 1 Television as an advertising medium has undergone significant advances since its 

22 inception in 1939. Modern advances in digital technology now allow viewers to be very 

23 selective in choosing programs to watch. Other advances in digital technology have led to such 

24 programming enhancements as a "tail" on a hockey puck, or an overlaid first down marker on 

25 a football field. The same technological advances allow improvements in the way 

26 advertisements are currently presented. Many sporting events are now presented with virtual 

27 object advertisements included in the video. For example, the broadcast of a major league 

28 baseball game may include one or more virtual object spots in which an advertisement is 
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1 displayed. The advertisements are then seen by television viewers, but not by fans who attend 

2 the game. However, the advertisements are assigned and insertedinto the program stream at 

3 a central location such as at a television broadcast facility. 

4 Summary 

5 AsystemandamemcddeUversvirtualobjectstorec^puonsitesortermm Avirtual 

6 object may be a realistic, synthetic replica of an actual object. The virtual object is viewable 

7 within video programming and may be combined with original video and audio to supplement 

8 or replace portions of the original video and audio content. Virtual objects may be overlaid on 

9 video, partially or entirely obscuring the underlying video. An overlaid object may be static in 

10 nature, such as agraphical icon or the like, or alternatively maybe dynamic, such as a video 

1 1 clip, animation, or scrolling alphanumeric characters, for example. Overlaid objects maybe 

12 limited spatially to a fixed portion of the video screen, limited temporally to a given time for 

1 3 display, limited by a combination of both location and time, or tied to a spatially changing 

14 portion of the screen that is moving with time. Alternatively, virtual objects may be added to 

15 and embedded within the actual video. Multiple virtual objects may be embedded in the video 

16 in a multi-layer fashion. The virtual object is indistinguishable from the other video content 

17 sharing the field of view. Virtual objects may be interactive in nature. That is, a viewer may 

1 8 select an object and the selection will initiate a process whereby a reception site sends a 

19 command to a location designated by the interactive virtual object to initiate some 

20 action. 

2 1 An operations center may process the video signal to allow for the insertion of virtual 

22 objects into the video. An object delivery center serves as a standalone or supplemental 

23 system to the operations center to deliver virtual objects independently of the video with which 

24 the virtual objects are to be associated. A delivery network includes any of a number of 

25 different delivery systems to support the delivery of video and virtual objects from the 

26 operations center and the object delivery center to a local insertion center, or directly to a 

27 reception site. The delivery network is also used to deliver video and virtual obj ects from the 

28 local insertion center to the reception site. The reception site receives the video and virtual 
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1 objects and associates the appropriate virtual objects with the video based on targeting 

2 algorithms. 

3 The reception site collects virtual object viewing information. The reception site may 

4 use the virtual object viewing information and other information stored at the reception site to 

5 adjust selection of the virtual objects at the reception site. In particular, the reception site may 

6 be provided a virtual object retrieval plan that indicates specific virtual objects or categories 

7 of virtual objects to be inserted into the virtual object locations. The reception site, or terminal, 

8 compares the retrieval plan to virtual objects stored in memory, and chooses that virtual object 

9 for display in a particular virtual object location that best satisfied the criteria of the retrieval 

1 0 plan. The terminal may use virtual objects watched data, programs watched data, or any other 

1 1 viewer-related data, to adjust the outcome of the comparison process. 

12 A virtual object location defmer system determines where in the content the virtual 

1 3 objects are to be placed and the rules associated with their placement. Content may be video 

14 programming, commercials and advertisements, or electronic program guide (EPG) information, 

1 5 for example. A virtual object selector system determines those available virtual objects suitable 

1 6 for placement in a virtual object location. A targeted virtual object management system 

1 7 determines which reception sites or reception site groups should receive and display which 

1 8 virtual object for a given virtual object location. The virtual objects and targeting information 

19 are then distributed to reception sites. 

20 After the reception site receives and stores the virtual objects, the reception site will 

2 1 place the virtual objects into virtual object locations. The placement maybe based on virtual 

22 objects watched data and other user information. The reception site, or terminal, will display 

23 the combined content with the overlaid or embedded virtual object. 

24 The reception site stores information indicating that a virtual object was inserted. The 

25 accumulated history information maybe collected from the reception site at a later time for 

26 review purposes. The unique reception site identification information may also be provided 

27 with the collected data. As mechanisms become available to identify specific viewers in a 

28 household, the system will allow for individual identification information to also be provided with 
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1 collected data. Finally, after collection of the reception site viewing history data, the reception 

2 site returns used reception site memory space to the reception site. 

3 A central operations center can determine virtual object locations available for virtual 

4 object placement. Alternatively, a local insertion center can determine the virtual object 

5 locations. The operations center can determine the specific virtual objects to be placed in a 

6 virtual object location. Alternatively, the local insertion center may determine the specific virtual 

7 object to be placed in a virtual object location. The reception site itself can determine which 

8 virtual object is to be placed in a virtual object location based on its own internal 

9 routines. 

1 0 Content, virtual objects, and associated targeting / virtual object placement control can 

11 be relayed to reception sites and information extracted from the reception site. The reception 

12 site may reside within a digital cable set top box that has access to a delivery network. 

1 3 Alternately, the reception site may be components of digital television satellite receivers. The 

1 4 reception site may be incorporated into the circuitry of a television, thereby eliminating the need 

15 for a separate control device attached to the television. Alternatively, the reception site may 

16 be incorporated into a personal computer, personal data device, smart phone with a display, 

17 or electronic book device. 

1 8 Description Of The Drawings 

1 9 The detailed description will refer to the following drawings in which like numerals refer 

20 to like items, and in which: 

21 Figure 1 is an overview of the virtual object targeting delivery system; 

22 Figure 2 provides a pictorial representation of virtual objects and virtual object 

23 locations; 

24 Figure 3 is an example of an overlaid virtual object; 

25 Figure 4 is an example of an embedded virtual object; 

26 Figure 5 depicts an operations center; 

27 Figure 6 depicts a virtual object definer; 

28 Figure 7 is a pictorial representation of a virtual object location matte; 
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1 Figure 8 depicts a virtual object selector; 

2 Figure 9 depicts a targeted virtual object management system; 

3 Figure 10 shows configuration and set-up steps associated with targeting virtual 

4 objects; 

5 Figure 1 1 shows a subscriber information database system; 

6 Figure 12 shows a configuration set-up system; 

7 Figure 13 shows a virtual object targeting system; 

8 Figure 14 presents an embodiment of the overall process for assigning targeted virtual 

9 objects; 

1 0 Figure 1 5 presents an embodiment of a process used by the virtual object placement 

1 1 engine to assign virtual objects to virtual object locations; 

1 2 Figure 1 6 presents an alternate embodiment used by the virtual object placement engine 

13 to assign virtual objects to virtual object locations; 

14 Figure 17presentsyetanotherembodimentusedbythe virtual object placement engine 

15 to assign virtual objects to virtual object locations; 

16 Figure 18 shows functions of an alternate virtual object targeting routine; 

1 7 Figure 1 9 shows an embodiment of a matrices processing subroutine that is called by 

18 the virtual objects targeting sequence; 

1 9 Figure 20 shows a subroutine used to select the final groupings of virtual objects to be 

20 sent to the reception sites or group of reception sites; 

21 Figure 21 shows a representation of reception site groupings; 

22 Figure 22 shows an example of a division of available bandwidth; 

23 Figure 23 shows an alternative software program flow for an object targeting routine; 

24 Figure 24 depicts an object delivery center; 

25 Figure 25 presents embodiments associated with the delivery of virtual objects over a 

26 coaxial or fiber cable system to a reception site; 

27 Figure 26 presents embodiments associated with the delivery of virtual objects over a 

28 wireless broadcast system to a reception site; 
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1 Figure 27 presents embodiments associated with the delivery of virtual objects over a 

2 satellite broadcast system to a reception site; 

3 Figure 28 presents embodiments associated with the delivery of virtual objects over a 

4 wired data network to a reception site; 

5 Figure 29 presents embodiments associated with the delivery of virtual objects using 

6 the public switched telephony network (PSTN) to a reception site; 

7 Figure 30 presents embodiments associated with the delivery of virtual objects using 

8 wireless personal communications system (PCS) to a reception site; 

9 Figure 3 1 depicts several embodiments associated with the delivery of virtual objects 

10 using a national or local television broadcaster's signal; 

1 1 Figure 32 depicts a local insertion center; 

12 Figure 33 depicts an example of a reception site; 

13 Figure 34 depicts a local data collection center; 

14 Figure 35 depicts a central data collection center; 

15 Figure 36 depicts an interactive object servicing center; and 

16 Figure 37 presents processing performed by an interactive object servicing center. 

17 Detailed Description 

18 An overview of the virtual object delivery system is depicted in Figure 1. An 

1 9 operations center 1 0 performs the processing of a video content signal to allow for the insertion 

20 of virtual objects into the content 36. An object delivery center 15 serves as a standalone or 

2 1 supplemental system to the operations center 1 0 to deliver virtual obj ects independent of the 

22 content with which the virtual objects are to be associated. A delivery network 1 1 includes any 

23 of anumber of different delivery systems to support the delivery of the content 36 and virtual 

24 objects from the operations center 10 and the object delivery center 15 to a local insertion 

25 center 20 or directly to a reception site 30. A delivery network 1 2 is used to deliver content 

26 and virtual objects from a local insertion center 20 to the reception site 30. The reception site 

27 30 may be any device or terminal capable of receiving video, including a set top terminal, a 
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1 television, a personal computer, a wireless telephone, a wired telephone, a PDA device or any 

2 similar device or terminal. 

3 The reception site 30 receives the content 36 and virtual objects and associates the 

4 appropriate virtual objects with the content 36 based on targeting algorithms. The reception 

5 site 30 may collect virtual object viewing information and make the viewing information 

6 available to a local data collection center 40 or a central data collection center 50 using a 

7 delivery network 1 3 . Alternatively, the reception site 30 may retain all virtual object viewing 

8 information and use the information to target virtual objects locally without control from the 

9 operations center 10. The local data collection center 40 provides information collected from 

1 0 the reception site 30 to the local insertion center 20 to assist in the targeting of virtual objects. 

1 1 The central data collection center 50 provides information collected from the reception site 30 

12 to the operations center 10 to assist in the targeting of virtual objects. The reception site 30 

1 3 provides interactive requests, which are driven by the selection of interactive virtual objects, 

14 to a interactive object servicing center 60 using a delivery network 14. Interactive responses 

15 are returned by the interactive object servicing center 60 to the requesting reception site 

16 30. 

17 Virtual objects maybe realistic, synthetic replicas of actual objects. Virtual objects 

1 8 may also be caricatures of actual individuals, photographs or other life-like renderings of actual 

1 9 individuals, cartoon figures, text objects, graphical renderings, or icons, for example. The virtual 

20 objects may be animated or fixed. The virtual objects are combined with video and audio to 
2 ] supplement or replace portions of video and audio in original content 36. As shown in Figure 

22 2, the reception site 30 may contain or be connected to a display 35 on which the content 36 

23 may be displayed. An opportunity, advertisement spot, or location, in the content 36 that is 

24 available for the placement of the virtual object will be denoted as a virtual object location 37 

25 henceforward. Within the virtual object location 37, one or more individual virtual objects may 

26 be assigned, each denoted as a virtual object 38 henceforward. Multiple virtual object 

27 locations, shown as virtual object locations 37 and 39 may be present in the content 36. 
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1 Multiple virtual objects, shown as virtual objects 38 and 40 maybe present within the virtual 

2 object locations. 

3 As shown in Figure 3, virtual objects maybe overlaid on video, partially or entirely 

4 obscuring the underlying video. An overlaid virtual object may be static in nature, like a 

5 graphical icon, as shown by virtual object 42. Alternatively the overlaid virtual object may be 

6 dynamic, like a video clip, animation, or scrolling alphanumeric characters as shown by virtual 

7 object 44. Overlaid virtual objects maybe limited spatially to a fixed portion of the video, 

8 limited temporally to a given time for display, or limited by a combination of both location and 

9 time. Overlaid virtual objects may also be tied to a spatially changing portion of the video that 

10 is moving with time. 

1 1 Alternatively, as shown in Figure 4, virtual obj ects may be added to and embedded 

1 2 within video. In this alternative, the synthetic virtual object 38 could be indistinguishable from 

13 the other video content 36 sharing the field of view as shown by virtual object 46 and virtual 

14 object 48. For instance, today's technology allows for the virtual placement of a billboard at 

1 5 televised sports events and the placement of a virtual first down marker in televised football 

16 games. 

17 In an embodiment, virtual reality and animation technologies are combined with 

1 8 advanced digital video techniques to provide realistic interaction of virtual objects within video. 

1 9 Combining these technologies, a soda can may be synthetically placed in the video, and may 

20 then be made to change over time. This placement and subsequent modification can occur at 

2 1 the video's source, at an intermediate point within the distribution and delivery path, or at the 

22 reception site 30. Combining the placement of virtual objects with the ability to target specific 

23 virtual objects to specific viewers or groups of viewers allows one household to see a scene 

24 with the soda can for cola, while the next door neighbor sees a root beer soda can, for 

25 example. 

26 Virtual objects may be interactive in nature, where a viewer can select a virtual object 

27 35 and this selection will initiate a process whereby the reception site 30 sends a command to 

28 the location designated by the interactive virtual object 38 to initiate some action. Actions may 
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1 include linking to a Web site to display content related to the interactive virtual object 38, 

2 initiating a purchase transaction, or initiating a request for more information about the selected 

3 virtual object 38. 

4 The operations center 1 0 shown in Figure 1 may include a number of systems that act 

5 together in processing the content 36 for the inclusion of virtual objects, for the selection of 

6 appropriate virtual objects to be placed in the content 36, for the targeting of virtual objects to 

7 individual reception sites, and for the packaging and delivery of the content 36 and virtual 

8 objects to reception sites. 

9 Placement of virtual obj ects can be explicitly selected by the operations center 1 0, 

1 0 resulting in the specific selection and placement of virtual objects into content 36. Alternatively, 

1 1 the placement may be generically defined by the operations center 10. In this alternative, the 

1 2 reception site 30 performs all the processing associated with selecting the appropriate virtual 

1 3 object 3 8 to be placed in the content 36 based on basic guidelines provided by the operations 

14 center 10 and algorithms operating at the reception site 30. 

15 As shown in Figure 5 , the operations center 1 0 includes a virtual object location definer 

1 6 100, a virtual object selector 200, and a targeted virtual object management system (TVOMS) 

17 300. 

1 8 Figure 6 presents the virtual object location definer 100. A video capture processor 

19 110 processes video and audio content 36 on a frame by frame basis, converting the original 

20 content 36 into a corresponding digitized representation. The processed content 36'is then 

21 stored in content buffer 120 for future access. A pre- viewer subsystem 130 allows for the 

22 viewing of a video frame of the processed content 36'. Frame N 1 4 1 , for example, (shown in 

23 Figure 7) associated with the processed content 36', may be retrieved from the content buffer 

24 1 20, viewed, and passed to a location selector processor 140. The location selector processor 

25 140 allows for the selection ofwhere in the frame N 141 the virtual object 38 maybe placed. 

26 When the frame N 141 is retrieved by the location selector processor 140, either a static area 

27 may be selected, or alternatively, a dynamic area, which is tied to an area within the frame of 

28 the processed content 36', may be selected. An overlay matte 1 6 (see Figure 7) may be used 
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1 in the virtual object insertion process to identify where and how a virtual object location 37 is 

2 to be placed in the processed content 36'. 

3 Techniques for pattern recognition used by the location selector processor 1 40 to 

4 facilitate the creation of the matte 16 and the identification of the pixels within the frame that the 

5 matte 1 6 is to be associated with for that frame are described in detail in US Patent 5,808,695 , 

6 to Rosser, Roy J.; Das, Subhodev; and Tan, Yi; entitled Method of Tracking Scene Motion 

7 for Live Video Insertion; US Patent 5,903,3 17, to Sharir, Avi; and Tamir, Michael; entitled 

8 Apparatus and method for Detecting, Identifying, and Incorporating Advertisements in a Video; 

9 US Patent 5,524,065, to Yagasaki, Toshiaki; entitled Method and Apparatus for Pattern 

10 Recognition; US Patent 5,627,915, to Rosser, Roy J.; Das, Subhodev; and Tan, Yi; von 

1 1 Kaenel, Peter; entitled Pattern Recognition System Employing Unlike Templates to Detect 

1 2 Objects Having Distinctive Features in a Video Field; and US Patent 4,8 17, 17 1 , to Stentiford, 

13 Frederick; entitled Pattern Recognition System, the disclosures of which are hereby 

14 incorporated by reference. 

1 5 When the area is selected by the location selector processor 1 40 and the overlay matte 

16 1 6 for the initial video frame N 141 is created, a video object marker processor 160 creates 

17 the transparent overlay matte 16 that is associated with the selected area for subsequent 

18 frames, for example frame N+l 142 and frame N+2 143 of the processed content 36', for the 

1 9 duration of frames designated, as shown in Figure 7. This selected area defines the virtual 

20 obj ect location 37 . Pattern recognition technology may then be applied to each subsequent 

21 frame of the processed content 36' in the video object marker processor 160, creating a 

22 sequence of mattes to be applied to each frame of the processed content 36', moving and 

23 transforming as needed to match the temporal movement and transformations of the virtual 

24 object location 37 within the processed content 36'to which the virtual object 38 is to be tied. 

25 The pattern recognition technology handles transitions, cutaways, and cutbacks within the 

26 processed content 36', and any visual blocking or occlusions that may occur as other objects 

27 within the processed content 36' appear in front of the dynamic area selected for virtual object 

28 location 37. 
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1 Simultaneously with the selection of the virtual object location 37 and the creation of 

2 the mattes, a virtual object rules processor 170 allows for the entry of rules that govern the 

3 types of virtual objects and other relevant placement guidelines associated with the virtual 

4 object location 37. These rules allow for the selection of characteristics such as the duration 

5 of the virtual object location 37, and viewing overlay characteristics such as transparency of 

6 the overlay virtual object. The operations center 10 processes the stored, non-realtime 

7 processed content 36'and the real-time (live) processed content 36' For real-time processed 

8 content 36'the content buffer 120 serves as a short buffer, and predefined rules are pre-loaded 

9 into the virtual object rules processor 170. Additionally, the video object marker processor 

10 1 60 is pre-loaded with the directions as to which locations within the processed content 36' are 

11 to be treated as virtual object locations. The video object marker processor 160 then 

12 automatically searches the real-time processed content 36' using pattern recognition 

1 3 technologies presented above, or other technologies, and automatically creates the mattes 

14 required for each virtual object location. Once the video object marker processor 1 60 creates 

15 the mattes and the associated controls, the mattes are associated with the actual processed 

16 content 36'in the content buffer 120. The processed content 36', along with the mattes are 

17 then optionally processed using the optional video processor 150, which performs any 

18 necessary content encoding (e.g., MPEG4, or digitalization), and makes the content 36' 

1 9 available to a rules application processor 1 80. The rules application processor 1 80 creates 

20 metadata packets that carry the virtual object placement rules information and mattes and 

2 1 associates these packets with the processed content 36' for each virtual object location 37 

22 selected in the virtual object location definer 100. 

23 Figure 8 is a block diagram of the virtual object selector 200. Processed content 36', 

24 along with the metadata packets carrying the virtual object placement rules information 

25 associated with each virtual obj ect location 37 and the mattes 1 6 are provided by the virtual 

26 object location definer 100 to the virtual object selector 200. An object selector processor 

27 210 extracts the placement rules and stores the processed content 36'in a content buffer 240. 

28 Using the placement rules, along with any operator entered object placement guidance, the 
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1 object selector processor 210 queries an object matcher processor 230 to initiate the selection 

2 of virtual objects that match the requisite rules. The object matcher processor 230 can be 

3 commanded by the object selector processor 2 10 to match a virtual object 3 8 in at least three 

4 manners: 1) automatically, 2) with manual placement, and 3) with pre-selected virtual objects. 

5 For automatic matching, the object matcher processor 230 searches an available virtual objects 

6 database 220 to find virtual objects that meet the placement rules provided by the object 

7 selector processor 210. The matching virtual objects are then marked in the available virtual 

8 objects database 220 as suitable for that virtual object location 37. For manual matching, the 

9 operator of the object matcher processor 230 manually selects the desired virtual objects to 

10 be associated with a virtual object location 37, and marks the selected virtual objects as 

1 1 suitable for the virtual object location 37 in the available virtual objects database 220. For 

12 pre-selected objects, the placement rules will indicate the pre-defined virtual objects to be 

13 associated with the processed content 36'. The object matcher processor 230 marks the 

14 pre-determined virtual objects in the available virtual objects database 220 as being associated 

15 the particular processed content 36' and virtual object location 37. 

1 6 Virtual objects may be processed and stored in the available virtual objects database 

17 220 before they are used. Processing of the virtual objects includes digitizing the virtual object 

18 38 and associating the virtual object with those virtual object 38 placement guidelines and rules 

1 9 that must be followed to place the virtual object 38 within virtual object locations. The rules 

20 and guidelines may include product categories with which the virtual object 38 should be 

2 1 associated, or in contrast, cannot be associated with, the type of virtual object 3 8, the duration 

22 that the virtual object 3 8 is valid to be used, and the number of times the virtual object 38 may 

23 be used. 

24 In a non-realtime environment, an optional post viewer processor 260, which is 

25 preceded by a virtual object insertion processor 250, is used to view the content 36 and insert 

26 each virtual object 3 8 that was matched to the content 36 by the object matcher processor 230 

27 in the corresponding virtual object location 37. Techniques for insertion of overlaid virtual 

28 objects are described in detail in U.S . Patents 4,3 1 9,266 to B annister, Richard S . ; entitled 
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1 Chroma Keying System; 4,999,709 to Yamazaki, Hiroshi; and Okazaki, Sakae; entitled 

2 Apparatus for Inserting Title Pictures; 5,249,039, to Chaplin, Daniel J.; entitled Chroma Key 

3 Method and Apparatus; and 5,233,423 to Jernigan, Forest E.; and Bingham, Joseph; entitled 

4 Embedded Commercials within a Television Receiver using an Integrated Electronic Billboard, 

5 the disclosures of which are hereby incorporated by reference. 

6 Techniques for the insertion of embedded virtual objects are described in detail in U.S. 

7 Patents 5,953,076, to Astle, Brian; and Das, Subhodev; titled System and Method of Real 

8 Time Insertions into Video Using Adaptive Occlusion with a Synthetic Reference Image; 

9 5,892,554, to DiCicco, Darrell; and Fant, Karl; entitled System and Method for Inserting 

10 Static and Dynamic Images into a Live Video Broadcast; 5,5 1 5,485, to Luquet, Andre; and 

1 1 Rebuffet, Michel; entitled Method and Device for Modifying a Zone in Successive Images; 

12 5,903,317, to Sharir, Avi; andTamir, Michael; entitled Apparatus and Method for Detecting, 

13 Identifying and Incorporation Advertisements in a Video; and the MPEG4 standard, the 

14 disclosure of which are hereby incorporated by reference. 

15 In a realtime environment, the optional post viewer processor 260 is bypassed, and the 

1 6 default virtual object 38 is placed in the virtual object location 37 by a default virtual object 

1 7 insertion processor 270, which includes (not shown) a virtual object insertion processor 250. 

1 8 The targeted virtual object management system (TVOMS) 300 shown in Figure 9 

19 allows for virtual objects, including virtual object-based advertisements, to be directed to 

20 subscribers based on, for example, the use of subscriber data, programs watched data, past 

2 1 virtual objects viewing data, and/or mood indicators entered by the subscriber. Alternatively, 

22 input from subscribers collected through form-based questionnaires (hard copy, electronic, and 

23 telephone, for example) may be used to further define a subscriber's potential likes, wants, and 

24 needs. Advertisers wanting to optimize their advertising expenditures may direct virtual objects 

25 to the appropriate viewing audiences to ensure that specific virtual objects are viewed by the 

26 desired audience. Specifically, advertisers can display specific virtual objects in content 36 that 

27 is being viewed by those subscribers most likely to be influenced to buy the advertised product, 

28 or otherwise respond in a desired fashion to the virtual objects. 
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1 Virtual objects may also be targeted to reception sites on various levels. At a highest 

2 level, virtual objects can be delivered to all reception sites viewing content 36, with no targeting 

3 of the virtual objects to the subscriber, but with the virtual objects displayed in the content 36 

4 that are determined to be most relevant to the content 36. That is, the virtual objects are 

5 placed in the virtual object location 37 without the use of an individual or group targeting 

6 algorithm. Alternatively, some level of targeting may occur based on, for example, ADI, zip 

7 code 44, geographical data and other similar criteria. In this alternative embodiment, the virtual 

8 objects are sent to a terminal, and a local insertion routine in the terminal controls placement 

9 of the virtual objects into the virtual object locations 37 in the content 36. The virtual objects 

1 0 may be stored in the terminal and may be periodically refreshed. To account for terminals that 

1 1 do not have virtual objects available for insertion, the content 36 may be provided with a 

1 2 default virtual object embedded in the content 36. Upon receipt of the content at a terminal, 

1 3 the terminal, using the local insertion routine, determines if the default virtual object should be 

14 replaced with another virtual object residing in the terminal's memory. 

1 5 Alternatively, virtual obj ects may be targeted to groups of reception sites, with the 

1 6 groups of reception sites categorized based on some other common subscriber characteristics 

1 7 such as programs watched data, for example. Finally, virtual objects may also be targeted to 

1 8 specific subscribers that share the use of a reception site 30 based on their unique subscriber 

19 characteristics. 

20 To target virtual objects, the TVOMS 300 may make use of information from 

2 1 numerous sources. These sources include collected programs watched data that are stored in 

22 the reception site 30, and periodically uploaded to the central data collection center 50 or the 

23 local data collection center 40, and from past virtual objects viewed information that is stored 

24 in the reception site 30 and periodically uploaded to the data collection centers. Additionally, 

25 these sources may include information from marketing databases and past television programs 

26 watched data, as described in U.S. Patent No. 5,798,785, entitled TERMINAL FOR 

27 SUGGESTING PROGRAMS OFFERED ON A TELEVISION PROGRAM DELIVERY 

28 SYSTEM, filed December 2, 1993, incorporated herein by reference. 
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1 The TVOMS 300 supports the management of information required to support each 

2 of the following: (1) delivery of targeted virtual objects along with content 36 being broadcast; 

3 (2) delivery of targeted virtual objects to subscribers independent of any content 36 being 

4 broadcast; and (3) delivery of TVOMS-related subscriber-specific information and commands. 

5 Figure 9 shows the TVOMS 300 supporting the targeting of virtual objects to 

6 subscribers. Broadcast information can be destined for the entire population of subscribers 

7 receiving the content 36, groups of subscribers, and individual subscribers. Broadcast 

8 information can include actual content 36, metadata packets with virtual object insertion control 

9 information, virtual objects for placement within the content 36, and command information 

1 0 required by the subscriber's reception site 30 to configure the reception site 30 and place the 

1 1 appropriate virtual object 38 within the content 36. Broadcasting may be supported over a 

1 2 variety of broadcast-capable communication systems, such as the Internet, cable television 

13 systems, terrestrial broadcast systems, satellite broadcast systems, and wireless 

14 communications systems, and other systems described below. 

15 A subscriber information database 1210contaimsubscriberMormationcollectedfrom 

1 6 numerous sources for each subscriber or reception site 30. The subscriber information may 

1 7 then be used by a virtual object targeting system 1 220 to determine the best virtual obj ects to 

18 be distributed for inclusion in the content 36. Additionally, the information collected may be 

1 9 used to determine if the subscriber information has changed to the point that refreshed virtual 

20 objects should be delivered to a subscriber or, alternatively, whether a subscriber's group 

2 1 assignments should be updated. The virtual object targeting system 1 220 determines the 

22 optimum subset of virtual objects to be associated with the content 36 based on the selected 

23 object metadata provided by the virtual object selector 200 (Figure 5) and subscriber 

24 information from the subscriber information database 1210. A content and virtual object 

25 packager 1 260 is directed to retrieve the appropriate virtual objects from an available virtual 

26 objects database. The content and virtual object package 1 260 then, along with the content 

27 36, from a content buffer 1270, addresses the virtual objects with the appropriate group 

28 addressing information, and packages the virtual objects with the content 36. A delivery 
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1 packager 1300 then delivers the combined package of virtual objects, content 36, and 

2 metadata to subscribers. 

3 As an alternative to delivering virtual objects with associated content 36, virtual objects 

4 can be delivered independently to individual subscribers or groups of subscribers based on 

5 updated subscriber information, modified group assignments, or the need for refreshed virtual 

6 objects at the reception site 30. Initiation could be automatic based on a scheduled cycle or 

7 by TVOMS operator direction. Upon delivery initiation, the virtual object targeting system 

8 1 220 uses subscriber information from the subscriber information database 1210, information 

9 about available virtual objects from the available virtual objects database 1265, and information 

10 about previously delivered virtual objects from the subscriber information database 1 2 10, to 

1 1 select the appropriate virtual objects to be packaged and delivered to a reception site 30. 

1 2 Once the virtual object targeting system 1220 determines the appropriate virtual objects, the 

1 3 content and virtual object packager 1260 retrieves the appropriate virtual objects, packages 

14 the virtual objects with reception site configuration information, addresses the information either 

15 to a single subscriber or group of subscribers, and delivers the information to the appropriate 

16 reception site 30 using a delivery processor 1 300. This delivery can be done in broadcast 

17 fashion or by communicating to reception sites directly. Virtual objects may alternately be 

1 8 broadcast to all reception sites, and a reception site 30 may store only the virtual objects that 

19 are associated with groups to which the reception site 30 belongs. Alternatively content 36, 

20 virtual objects, and other information destined to reception sites may be provided to the object 

21 delivery center 15 (Figure 1) for delivery. 

22 The databases addressed in Figure 9 may be configured to support a variety of 

23 information necessary for the TVOMS 300 to manage the targeting process. Below are tables 

24 that present typical data that may be tracked by these individual databases. 

25 Subscriber Information Database 1210 

26 Reception system identification information 

27 Reception site type 

28 Date of system set-up 
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1 


Date of last communication with operations center 


2 


Household income 


3 


User data (for each registered subscriber), including: 


4 


Name 


5 


Sex 


6 


Age 


7 


Place of birth 


8 


Education 


9 


Profession 


10 


TV program preferences 


11 


Demographic information 


12 


Past advertising viewed data which virtual obiects. time snent viewing. 


13 


Past products ordered, along with time, date, and method of order 


14 


Past billing information 


15 


ImDuted subscriber data from marketing databases 


16 


Past TV programs watched data, along with time and date 


17 


Past PPV programs ordered data, along with time and date 


18 


Mood indicators 


19 


Form based questionnaire results 


20 


Communication methods available (available options for both return and 


21 


delivery) 


22 


Group assignments per subscriber for each category 


23 


Past virtual obiects delivered to subscriber date of delivery* method of 


24 


delivery 




Zip+4 information 


26 


Available Virtual Objects Database 1265 


27 


Virtual object identifier with actual digital version of virtual (CR) object Display 


28 


options (e.g., text, audio, graphics, video, link, HTML, XML, interactive) 
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1 Static vs. dynamic virtual object indicator, 

2 If a linked virtual object, link table information 

3 Pricing subsidy information 

4 Run through completion status mode indication 

5 Date of valid use 

6 Virtual object placement controls, acceptable frequency 

7 Category and group preferences (as virtual object ranking percentages) 

8 Pending Commands Database 1215 

9 For each pending command: 

10 Destination address 

1 1 Actual command 

12 Date generated 

13 Date of confirmed receipt 

14 Within the TVOMS 300, the virtual object targeting system 1220 is responsible for the 

1 5 intelligent and rapid selection of virtual objects for placement in content 36 . Category and 

1 6 group targeting is managed in a manner similar to that described in co-pending U.S . Application 

17 Serial No. 09/054,4 1 9 entitled TARGETED ADVERTISEMENT USING TELEVISION 

18 DELIVERY SYSTEM, filed April 3, 1998, and in co-pending U.S. Application Serial No. 

19 09/328,672 entitled ELECTRONIC BOOK SELECTION AND DELIVERY SYSTEM 

20 WITH TARGETED ADVERTISING, filed on June 9, 1 999, both of which are incorporated 

2 1 herein by reference. 

22 Careful management of the virtual objects within the content 36, based on information 

23 known about the demographics and viewing habits of subscribers, for example, can greatly 

24 increase both the advertisers' likelihood of reaching an interested subscriber, and the likelihood 

25 a subscriber will be interested in a specific virtual object 38. Each virtual object location 37 

26 within the content 36 is assigned a series of virtual objects by the TVOMS 300, and when 

27 multiple virtual objects are delivered for a given virtual object location 37 in the content 36, a 

28 retrieval plan is developed that directs which virtual objects should be displayed for a given 
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1 subscriber or reception site 30, a group of subscribers or reception sites, or the entire 

2 subscriber population. 

3 The process of managing the targeted virtual objects begins with a number of 

4 configuration and set-up steps shown in Figure 1 0 that begins with the start step shown in 

5 block 7010 and ends with the end step shown in block 7017. First, individual reception site 

6 address information is collected by a subscriber data collection engine 1 202 in the address 

7 information collection block 7011. This address information uniquely identifies each reception 

8 site 30 subscriber and associates necessary address information about each subscriber with the 

9 reception site identifier to aid in the virtual objects targeting process. This address information 

1 0 includes subscriber profile information, programs viewed information, past virtual objects 

1 1 delivered and viewed, and responses to menu-based questionnaires or other questionnaires 

1 2 completed by the subscriber. In block 70 1 2, other subscriber information may be collected 

1 3 from various sources, including surveys and marketing databases correlated by address or zip 

14 code+4, for example. 

1 5 Next, a number of target categories are defined as shown in block 7013. Examples 

16 of target categories include demographic targeting (age/sex/income) and location, such as Area 

17 of Dominant Influence (ADI). Next, as shown in block 7014, each target category is then 

1 8 segmented into appropriate groups . For example, the ADI may include Los Angeles, C A and 

19 Washington D.C. New target categories can be added and the groups redefined after their 

20 initial establishment. 

21 Next, as shown in block 7015, for each target category, each reception site 30 is 

22 assigned to a group based on the information collected about the subscriber. Once each 

23 subscriber is assigned to a group, the group assignments are conveyed to the reception site 30 

24 and stored therein, as shown in block 70 1 6. As groups are modified or group assignments 

25 change, the reception sites are provided with the changes. Additionally, the group assignment 

26 information is periodically resent to the reception sites to ensure that newly added reception 

27 sites and those reception sites that have accidentally lost their information are up-to-date. 

28 Alternatively, the reception site 30 may perform the processing of information about the 
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1 characteristics of the subscriber, and generation of the group assignment information internal 

2 to the reception site. 

3 Retailing to Figure 9, the virtual object targeting system 1220 determines the optimum 

4 types of virtual objects to be placed in the content 36 from the selected virtual objects provided 

5 bythe virtual object selector 200 (Figure 5). The virtual object targeting system 1220takes 

6 into account subscribers who will likely view the content 36, the desirability of providing 

7 available virtual objects to those subscribers, target categories, the number of virtual objects 

8 locations available for the content 36, and the number of virtual objects available for assignment 

9 for a given virtual object location 37. 

10 Once specific virtual objects are selected for one or more available virtual object 

1 1 locations 37, the groups that should view each virtual object 38 are determined, based on the 

1 2 target category of interest. The selected virtual object locations 37 may include all virtual 

1 3 object locations, or a subset of all the virtual object locations. Assignment of a reception site 

14 30 to a group for the appropriate virtual objects may be based on a detailed retrieval plan. The 

1 5 retrieval plan may provide information for one virtual object location 37 or multiple virtual 

1 6 object locations within content 36, where one or more virtual objects, target categories, and 

17 the groups to which each virtual object 38 is targeted within each virtual object location 37 is 

1 8 also provided. An example retrieval plan is provided in Table C below. Alternatively, the 

19 retrieval plan providing virtual object assignments to virtual object locations may be sent 

20 independently from the retrieval plan providing virtual objects, target categories, and the groups 

21 to which each virtual object 38 may be targeted. Retrieval plans may be distributed along with 

22 the virtual objects and the associated content 36 directly to the reception sites by the delivery 

23 processor 1 300 or using the object delivery center 1 5 . Alternatively, a retrieval plan may be 

24 distributed by the delivery processor 1 300 or using the object delivery center 1 5 independent 

25 of the associated content 36 or virtual objects. 

26 After the reception site 30 receives and stores the virtual objects and the retrieval plan, 

27 the reception site 30 inserts those virtual objects into the appropriate virtual object locations 

28 in the content 36 based on the retrieval plan. The reception site 30 may retrieve and store only 
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1 those virtual objects associated with thatreception site's group assignment forthat virtual object 

2 location 37 . Alternatively, the reception site 30 may retrieve and store all virtual objects but 

3 only insert those virtual objects into virtual object locations as dictated by the retrieval 

4 plan. 

5 When the virtual objects are displayed within the content 36, the reception site 30 will 

6 store virtual objects viewed data indicating that a virtual object 38 was shown. In an 

7 embodiment, the reception site 30 will store this virtual object viewed data only if the virtual 

8 objects are displayed for a predetermined time, or only if the subscriber takes an action to 

9 indicate the virtual object 38 has been viewed, such as by selecting an interactive virtual object 

10 38, for example. Accumulated virtual objects viewed data may be collected from a reception 

1 1 site 30 at a later time for review purposes. Unique reception site identification information also 

1 2 may be provided with the collected virtual obj ects viewed data. Upon collection of the virtual 

1 3 objects viewed data, the reception site 30 may return the used memory space to available 

14 pools for future use. 

15 The virtual object targeting system 1220 receives requests from the metadata extractor 

1 6 processor 1 200 to initiate the determination of virtual objects to be placed. The metadata 

1 7 extractor processor 1 200 receives content 36 and associated virtual object information from 

18 the virtual object selector 200 (Figure 5). The virtual object targeting system 1220provides 

19 outputs to the content and virtual object packager 1260 and the retrieval plan generator 1275. 

20 A part of the TVOMS 300 operation is the retrieval of subscriber data, and the 

2 1 assimilation of the subscriber data into the virtual objects selection method. This operation 

22 typically includes two steps. First, subscriber data is retrieved from the reception sites by the 

23 central data collection center 50 or the local data collection center 40 (Figure 1). The 

24 subscriber data is compiled and sent to the data collection engine 1202 in the operations center 

25 10. Once assembled at the TVOMS 300, the data is filtered for each application of the 

26 TVOMS 300. In an embodiment, the subscriber information database 1210 receives inputs 

27 from the subscriber data collection engine 1202 and a configuration set-up system 1205. The 
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1 subscriber information database 1210 provides outputs to the configuration set-up system 

2 1205, and the virtual object targeting system 1220. 

3 The data gathered includes: 

4 What products a subscriber purchased and when they were purchased, 

5 What Pay Per View (PP V) TV programs a subscriber purchased and when 

6 they were purchased, 

7 What television programming a subscriber has viewed, 

8 What virtual objects a subscriber viewed and for how long, and 

9 Subscriber profile information. 

1 0 Subscriber profile information maybe collected and stored for one or more subscribers 

1 1 for the purposes of virtual objects targeting. The subscriber profile may include demographic 

12 information that may be gathered in a number of ways. The reception site 30 builds the 

1 3 subscriber profile for each subscriber and stores the information in a memory file by subscriber 

14 name. The file may be uploaded to the central data collection center 50 or the local data 

1 5 collection center 40 and provided to subscriber data collection engine 1 202 periodically. 

1 6 Subscriber preference information may be collected using on screen menus at the reception 

1 7 site 30, including information such as name, sex, age, place of birth, place of lower school 

1 8 education, employment type, level of education, amount of television program viewing per 

1 9 week, and the number of television shows in particular categories that the subscriber watches 

20 in a given week such as, sports, movies, documentaries, sitcoms, amount of Internet use and 

2 1 favorite web sites, etc. Any demographic information that will assist the TVOMS 300 in 

22 targeting virtual objects may be used. 

23 In addition to demographic information gathered at the reception site 30, the subscriber 

24 profile can be compiled using other methods. For instance, subscriber information can be 

25 gathered using questionnaires sent by mail and subsequently entered in the subscriber 

26 information database 1210. 

27 As an alternative to gathering demographic data, a simulated subscriber profile can be 

28 generated using an algorithm that analyzes subscriber access history and subscriber habits . 
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1 Using test information generated from a statistically significant number of subscribers, the 

2 simulated subscriber profile algorithm estimates the subscriber's age, education, sex and other 

3 relevant information. The analysis then compares information about the subscriber, for example 

4 the subscriber's programs watched information, with that of the test group. An example of the 

5 type of information maintained for a subscriber profile is presented below. 

6 The subscriber profile data fields are an example of typical fields that can be used in 

7 the databases. Definitions of various fields are listed below. The primary purpose of profiling 

8 the subscriber is to acquire marketing information on the subscriber's likely response to 

9 available virtual objects. Ancillary information may be available including actual program 

10 selections or interactive virtual objects selections. Information tracked within the subscriber's 

11 profile includes: 

1 2 Subscriber ID A unique identifier generated by the system, one for 

13 each subscriber using a specific reception site. 

14 Reception site types Boolean field that identifies the type of reception site 

15 used. 

16 Reception site ID ID of the reception site. 

17 Hookup Date Date physical hardware is connected. 

18 A demographic profile maybe constructed for each subscriber from questionnaires or 

19 other sources. The following fields represent this demographic information: 

20 Subscribers Age 2-5 Boolean field if the household has subscribers 

21 between 2 and 5 years of age. 

22 Subscribers Age 6-11 Boolean field if the household has subscribers 

23 between 6 and 1 1 years of age. 

24 Subscribers Age 12-17 Boolean field if the household has subscribers 

25 between 12 and 17 years of age. 

26 Subscribers Age N 1 -N2 Boolean field if household has subscribers between 

27 Nl and N2 years of age. 

28 Income Annual household income. 
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1 Zip Code+4 Self-explanatory. 

2 Occupancy Number of subscribers in household. 

3 Highest Education Highest level of education of any subscriber in the 

4 household. 

5 Field of Use Personal, professional, educational, other. 

6 Profession Self-explanatory. 

7 Education Level Self-explanatory. 

8 These subscriber profile inputs may assist in the assignment of reception sites to groups 

9 for each target category. There are numerous variations to the field definitions listed above, 

1 0 such as different age groupings, for example. Other subscriber profile data fields may also be 

1 1 specified. 

1 2 Marketing information, such as the demographics of subscribers, maybe received from 

13 a central data collection center 50, a local data collection center 40, other external sources, or 

14 directly from the reception sites using the subscriber data collection engine 1202. Toeffectively 

1 5 manage the virtual obj ects targeting operations, marketing information, such as the existence 

16 of markets for certain products, may be provided to the TVOMS 300. The following 

17 examples of information maybe maintained in the subscriber information database 1210: 

1 8 subscriber demographic profile, subscriber buy information, and correlation of demographic 

1 9 information with buy information. The subscriber data collection engine 1 202 gathers the 

20 marketing information from the various sources and indexes the information for inclusion in the 

2 1 subscriber information database 1210. 

22 To maintain the subscriber information database 1210 within the TVOMS 300, a 

23 database server 1 190, communications server 1 191, subscriber workstation 1 192 or stations, 

24 or the suitable equivalents thereof, may be used, as depicted in Figure 1 1 . The database server 

25 1 190 supports saving database files, eventlogging, event scheduling, multi-subscriber services, 

26 database server services, and database security access. 

27 The communications server 1191 performs the following functions on database data: 

28 integrity check, filtering, processing, downloading to reception sites using the pending 
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1 commands database 1215, and uploading subscriber data from reception sites using the 

2 subscriber data collection engine 1202. The subscriber workstation 11 92 allows for operator 

3 viewing and entry of subscriber data into the subscriber information database 1210. 

4 Figure 12 shows an example ofthe configuration set-up system 1205 in more detail. 

5 An interface 1206 receives individual addressing information unique to reception sites. The 

6 interface 1 206 can include a workstation, such as the workstation 1 209, for example, from 

7 which an operator manually enters reception site information. Alternately, reception site 

8 Mormationcanbeautomaticallyenteredattheinterface 1206 by downloading from an off-site 

9 database, the Internet, a storage medium, such as a CD-ROM or a floppy disk, or by 

1 0 collecting the information directly from the individual reception sites using the subscriber data 

1 1 collection engine 1202 or provided by a central data collection center 50 or local data 

12 collection center 40. Aprocessor 1207processesthereceivedreceptionsiteinformationand 

13 organizes the information for use. For example, the processor 1207 may create a 

14 Category/Group Definition Matrix as presented in Table A and a Group Assignment Matrix 

15 as presented in Table B that can be used to target virtual objects to groups of reception sites 

16 or to an individual reception site 30. In an alternative embodiment, if subscriber information 

17 is available where multiple subscribers may share a reception site 30, a Group Assignment 

18 matrix may be created for each subscriber who shares the reception site 30. The 

1 9 Category/Group Definition and Group Assignment matrices will be described in more detail 

20 later. The Category/Group Definition and Group Assignment matrices and organized reception 

2 1 site information are then stored in a database 1 208 , and are periodically updated as reception 

22 site information, for example, changes. 

23 The information used by the processor 1207 to create a database of the 

24 Category/Group Definition and Group Assignment matrices includes, for example, the reception 

25 site identifier, subscriber identifier, zip code + 4 data, household income, and age and sex of 

26 the subscribers, for example. The information gathered by the configuration set-up system 

27 1 205 can come from a variety of sources including marketing databases, direct inputs from the 

28 subscribers, data collected by the subscriber data collection engine 1202, a central data 
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1 collection center 50, a local data collection center 40, and other sources. Once the data are 

2 collected, the processor 1207 will assign category numbers to certain types of the data. For 

3 example, the ADI could be assigned category 1 and household (HH) income could be assigned 

4 category 2. Next, the configuration set-up system 1205 creates a number of non-overlapping 

5 groups for each category. For example, ADI can be broken down into Seattle, WA, 

6 Washington D.C., Denver CO., Los Angles CA, etc. Similarly, HH income can be broken 

7 down into a number of income groups such as no income, 20-40K, 40-60K, 60- 1 20K, and 

8 over 120K. Then, the configuration set-up system 1205 assigns a "group mask representation" 

9 for each group within every category. The group mask representation may be simply a binary 

10 number that can be used to identify a particular group. Table A shows a completed 

1 1 Category/Group Definition matrix that could be used by the virtual object targeting system 

12 1 220 to assign targeted virtual objects to groups of reception sites or to individual reception 

13 sites. 

14 Table A - Category/Group Definition Matrix 



19 



Category 
Number 


Category 
Name 


Group 
Number 


Group Definition 


Group Mask 
Representation 






1 


Seattle, WA 


1000000000 


1 


ADI 


2 


Washington, D.C. 


0100000000 






3 


Denver, CO 


0010000000 






4 


Los Angeles, CA 


0001000000 






1 


No income 


1000000000 


2 


HH income 


2 


20-40K 


0100000000 






3 


40-60K 


0010000000 






4 


60-120K 


0001000000 






1 


Group a 


1000000000 






2 


Group b 


0100000000 


3 


Category x 


3 


Group c 


0010000000 






4 


Group d 


0001000000 






5 


Group e 


0000100000 






6 


Group f 


0000010000 
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1 The processor 1207 also creates the Group Assignment matrix. The Group 

2 Assignment matrix, shown in Table B, assigns to each reception site 30, for each category, its 

3 corresponding group number. Associated with each group number is the group definition and 

4 the group mask representation. For example, the reception site 30 identified by the address 

5 1231 1 is assigned group number 2 (i.e., Washington D.C.) for ADI, and group number 3 (i.e., 

6 40-60K) for household income. The Group Assignment matrix is updated periodically as 

7 categories and group definitions change, and as data related to individual reception sites or 

8 groups of reception sites change. Many other ways of organizing the information in a database 

9 for later use are possible. 

10 The configuration set-up system 1205 also delivers the group configuration (i.e., 

1 1 information specific to an individual reception site 30, from the Group Assignment matrix) to 

12 Table B Group Assignment Matrix 
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Address 


Target Category 


Group 
Number 


Group Definition 


Group Mask 
Representation 




ADI 


2 


Washington, DC. 


01000000000 




HH income 


3 


40-60K 


00100000000 


12311 


Category x 


5 


Group d 


00100000000 




ADI 


4 


LA 


00100000000 




HH income 


3 


60-120K 


00100000000 


12312 


Category x 


2 


Group a 


10000000000 




ADI 


3 


Denver 


00100000000 


12313 


HH income 


4 


60-80K 


00010000000 




Category x 


3 


Group b 


01000000000 



17 each reception site 30. For example, the reception site 30 assigned the address 12311 

18 is sent for category 1, group mask representation 01000000000, indicating group 2 

19 assignment. 

20 The group configuration information can be stored in the pending commands database 

21 1 2 1 5 to be transmitted directly to each reception site 30 periodically or the next time the 

22 reception site 30 establishes communications operations center 10. Each time a group 
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1 configuration message is generated, the message is stored in the pending commands database 

2 1215. 

3 Alternatively to the T VOMS 300 assigning the reception site 30 to individual groups 

4 for each category, the TVOMS 300 could deliver the group definitions and category definitions 

5 to the all reception sites. Each reception site 30 could then assign itself to the appropriate 

6 groups for each category based on internal processing algorithms. 

7 Figure 1 3 shows an embodiment of the virtual object targeting system 1 220 in more 

8 detail. A resource management engine 1 305 uses information from a metadata extractor 

9 processor 1200 and an available virtual object database 1265 (see Figure 9) to determine the 

1 0 number of virtual objects to be assigned to a given virtual object location 37. A virtual object 

1 1 placement engine 1307 decides which virtual objects to place in virtual object locations in the 

12 content 36. A group assignment engine 1309 determines which reception sites will view 

1 3 specific virtual objects. The virtual object placement engine 1 307 receives information from 

1 4 the resource management engine 1 305 related to the number of virtual objects available, how 

1 5 many virtual objects are to be provided for a given virtual object location 37, and the actual 

16 type of virtual objects available. 

17 The resource management engine 1305 functions to divide available delivery bandwidth 

1 8 among multiple virtual objects for a given virtual object location 37 in the content 36. Because 

1 9 there may be alimited amount of resources on the delivery network 1 1 to deliver virtual objects 

20 with the content 36, the resource management engine 1 305 may assign the available bandwidth 

2 1 optimally for the virtual objects associated with the individual virtual object locations within the 

22 content 36 being delivered over the communication channels. Some virtual object locations 

23 may be assigned multiple virtual objects, each targeted to a different group or groups, whereas 

24 other virtual object locations may be assigned only a single virtual object 38. 

25 Referring to Table A, four group numbers (i.e., 1 -4) are shown for the category of 

26 targeted virtual objects, ADL For a particular virtual object location 37 in the content 3 6, the 

27 four groups can be divided into two, one for each available virtual object 38 of two total, with 
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1 groups 1 and 2 receiving virtual object A and groups 3 and 4 receiving virtual object B, as 

2 shown for virtual object location 1. This later example is shown in Table C. 

3 Table C - Retrieval Plan 



Virtual Object 
Location 


Target 
Category 


Virtual Object To 
Retrieve 


Groups Assigned to 
Specific Virtual 
Object 


Group Mask 
Assignment 


Virtual Object Location 1 


ADI 


Virtual Object A 


1,2 


11000000000 






Virtual Object B 


3,4 


00110000000 


Virtual Object Location 2 


HH 


Virtual Object A 


1,2,3 


11100000000 




Income 


Virtual Object B 


4 


00010000000 


Virtual Object Location 3 


Category x 


Virtual Object A 


1,2 


11000000000 






Virtual Object B 


3 


00100000000 






Virtual Object C 


4 


00010000000 






Virtual Object D 


5 


00001000000 






Virtual Object E 


6 


00000100000 


Virtual Object Location 4 


All 


Virtual Object Al 


All 


11111111111 



10 After determining how many virtual objects will be needed for each virtual object 

1 1 location 37 within the content 36, the resource management engine 1 305 may also account for 

12 the type of available targeted virtual objects for display and the variety of subscribers 

1 3 (according to group assignment numbers) who may be viewing the content 36. An advertiser 

14 may provide this information when forwarding virtual objects for insertion, 

15 In an embodiment, the virtual object placement engine 1307 determines which specific 

1 6 virtual objects are to be placed in each available virtual object location 37 within the content 

17 36. The virtual object placement engine 1307 first receives the list of selected available virtual 

18 objects from the metadata extractor processor 1200 (Figure 9). In cooperation with the 

19 resource management engine 1305, the virtual object placement engine 1307 then determines 

20 which of the available virtual objects should be placed in each virtual object location 37 within 

2 1 the content 36. For example, if the preferred target category for virtual object location 1 is 
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1 ADI, the virtual object placement engine 1 307 will select one or more targeted virtual objects 

2 determined by the metadata extractor processor 1 200 to place in that virtual object location 

3 37. If the demographic or other data assembled by the configuration set-up system 1205 

4 indicates that more than one targeted virtual object 38 should be placed, depending on the 

5 ADI, then the virtual object placement engine 1 307 will select the appropriate number of 

6 targeted virtual objects, and will assign each targeted virtual object 38 to the specific virtual 

7 object location 37. The operation ofthe virtual object placement engine 1307 to assign the 

8 targeted virtual objects will be described in more detail later. 

9 In an embodiment, the group assignment engine 1 309 receives inputs from the resource 

10 management engine 1305 and the virtual object placement engine 1307 and then determines 

1 1 which reception sites will view specific targeted virtual objects. Thus, for each virtual object 

1 2 location 37, the group assignment engine 1 309 assigns the reception sites to one ofthe virtual 

1 3 objects. The reception sites can be assigned based on their placement within a group (i.e. , 

14 based on their group assignment number) or based on their individual reception site unit 

15 address. In tables B and C, the assignments are shown based on the group assignment 

1 6 numbers. As also shown in Table C, the group addressing for a virtual object location 37 may 

17 be based on a single category of targeting. This may avoid a conflict regarding which virtual 

18 object 38 a reception site 30 may retrieve. 

1 9 The group assignment engine 1 309 provides an output to the retrieval plan generator 

20 1275. The output indicates which group assignment numbers (i.e., which groups of reception 

2 1 sites) are assigned to a virtual object 38 for a given virtual object location 37 in the content 36. 

22 The retrieval plan generator 1 275 then generates a bit word, or group mask assignment, that 

23 is used to assign the groups to virtual objects. Once generated, the retrieval plan is provided 

24 to the delivery processor 1 300 for distribution along with the content 36 and the actual virtual 

25 objects to reception sites by object delivery center 15. 

26 In an embodiment, the virtual object targeting system 1220 provides a virtual object 

27 generation request command 1261 to the content and virtual object packager 1260. The 

28 virtual objects generation request command 126 1 specifies which particular virtual objects are 
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1 to be displayed in a particular virtual object location 37, and the actual location of the virtual 

2 objects. The virtual object 38 is then retrieved from the available virtual object database 1265. 

3 The virtual objects, along with the retrieval plan, and content 36 and associated metadata 

4 packets are provided to the delivery processor 1 300 for delivery to the appropriate reception 

5 sites. 

6 When a reception site 30 receives the content 36 that contains targeted virtual objects, 

7 software instructions operating on the reception site 30 analyze the contents of the retrieval 

8 plan. Then, based on the groups assigned for each virtual object 38, the reception site 30 

9 retrieves those virtual objects that match its own group assignments for the target category 

1 0 being used for the virtual object location 37. The reception site 30 then associates those virtual 

1 1 objects retrieved with the appropriate virtual object location 37 where the virtual object 38 will 

12 be placed, so that when the content 36 is viewed, the virtual object 38 assigned to that virtual 

13 object location 37 is displayed. 

14 An embodiment of the process for assigning targeted virtual objects using the virtual 

1 5 object placement engine 1 307 is presented in Figure 14. The process begins with block 2360. 

1 6 In block 2362, the virtual object placement engine 1 307 assigns reception sites to groups. In 

17 block 2364, the virtual object placement engine 1307 ties or relates virtual object locations in 

1 8 content 36 to the groups. In block 2366, the virtual object placement engine 1307 ties or 

19 relates virtual objects to groups. In block 2368, the virtual object placement engine 1307 

20 determines how many virtual objects to assign to a virtual object location 37 . In block 2370, 

2 1 the virtual object placement engine 1 307 determines which target category to use for one ore 

22 more virtual object locations 37. In block 2372, the virtual object placement engine 1 307 

23 determines specific virtual objects to be placed in the virtual object locations 37. In block 

24 2374, the virtual object placement engine 1 307 determines which groups to assign to the virtual 

25 objects 38 for the selected virtual object locations 37. The process ends with block 

26 2376. 

27 As discussed in the above embodiment, virtual object targeting uses target categories 

28 and groups within each target category to tie or relate three entities together: 1) the reception 
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1 site 30; 2) virtual objects; and 3) virtual object locations in content 36. In one embodiment of 

2 block 2362 in Figure 14, the reception sites are assigned to groups for each target category by 

3 the configuration set-up system 1 205 based on numerous factors as described below. One 

4 method to assign the reception sites to groups is to use the zip code+4 as an index into one of 

5 the available demographic marketing databases. From the zip code+4 data, a distinct 

6 demographicclustercanbedetermined. The demographic cluster can then be mapped directly 

7 to the specific group within each target category. Manual assignment of groups to reception 

8 sites would be a daunting task for a large population of reception sites (approaching several 

9 million). Therefore, the processor 1207 in the configuration set-up system 1205 may perform 

1 0 this function automatically, using its installed software routines. Alternative methods can also 

1 1 be devised to automatically map individual reception sites to groups within target categories. 

12 Once each reception site 30 is mapped to one group for each target category, the group 

13 assignments may be delivered to the reception site 30 for storage. 

14 hi one embodiment of block 2364 in Figure 14, virtual object locations in content 36 

15 are tied or related to groups as described below. For each virtual object location 37, a group 

1 6 breakdown percentage can be defined for each group that represents the likely compatibility 

17 of the content 36 surrounding that virtual object location 37 with each group. Breakdown 

1 8 percentages for each virtual object location 37 are defined within the virtual object selector 200 

1 9 (see Figure 8) and passed to the TVOMS 300. Table D shows a sample breakdown of these 

20 group breakdown percentages for five example virtual object locations for three example target 

21 categories. 

22 The group breakdown percentage data may be derived from a number of sources 

23 including surveys, ratings services, and virtual objects viewed data collected by the reception 

24 sites, for example. In this example, the three target categories are the same as those presented 

25 in Table B , and the group assignment numbers are the same as those presented in Table A. 

26 Thus, target categories 1 and 2 each have four groups associated with them, and target 

27 category 3 has six groups associated with it. For virtual object location 1 , the target category 

28 1 refers to ADI and under group 1 , a group breakdown percentage of 25 percent is assigned 
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1 for group 1 from the target category ADI since 25 percent of the subscribers reside in the 

2 Seattle, WA ADI. The group breakdown percentages for each target category for each virtual 

3 object location 37 may sum to 100 percent. 

4 In an embodiment of the relating subroutine represented by block 2366 of Figure 14, 

5 virtual objects may be ranked according to their potential revenue generation for each group 

6 within one and up to all possible target categories, again using percentages. This information 

7 may be provided by an advertiser, programmer, or content provider responsible for the virtual 

8 objects and may reside in the available virtual objects database 1265. Table E shows a sample 

9 assignment of virtual object ranking percentages for eight sample virtual objects using the same 

1 0 target categories and group numbers as in Table D. Not all virtual objects may be assigned to 

1 1 groups for a target category if an advertiser or programmer does not wish its virtual objects to 

12 be targeted in the manner required by that target category. For example, an advertiser or 

13 programmer may want the same virtual object to be displayed at all reception sites 30, 

14 regardless of identical subscriber or group information or characteristics. 



15 Table D - Virtual Object Location Group Breakdown Percentages 



16 
17 
18 


Virtual 
object 
location 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


19 
20 
21 


Virtual 
object 
location 1 


1 


25 


25 


25 


25 


N/A 


N/A 


22 


a 


2 


30 


10 


20 


40 


N/A 


N/A 


23 


a 


3 


10 


20 


30 


40 


N/A 


N/A 


24 
25 
26 


Virtual 
object 
location 2 


1 


10 


20 


30 


40 


N/A 


N/A 


27 




2 


25 


25 


25 


25 


N/A 


N/A 


28 




3 


10 


15 


25 


25 


15 


10 


29 
30 
31 


Virtual 
object 
location 3 


1 


40 


30 


20 


10 


N/A 


N/A 


32 




2 


80 


10 


5 


5 


N/A 


N/A 
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2 
3 
4 

5 

6 

7 
8 
9 

10 

11 
12 

13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 

24 

25 
26 
27 
28 
29 
30 
31 
32 



Virtual 
object 
location 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 




-x 


25 


25 


10 


10 


15 


25 


Virtual 
v ii Luai 

object 
location 4 


I 


50 


0 


50 


0 


N/A 


N/A 


u 


2 


o 


40 


40 


20 


N/A 


N/A 


u 




10 


10 


25 


25 


15 


15 


Virtual 

V 11 LU&1 

object 
location 5 


I 


20 


30 


30 


20 


N/A 


N/A 




2 


30 


30 


10 


30 


10 


10 




3 


10 


30 


10 


30 


10 


10 



Referring to Table E, the data indicates that for virtual object 1, and target category 1 
(ADI), the advertiser believes that virtual object 1 is appropriate for the subscribers in groups 
1 and 2 and is not appropriate for the subscribers in groups 3 and 4. The advertiser also 
believes that virtual object 1 is equally appropriate for both the group 1 and the group 2 
subscribers. However, if the group 1 subscribers are determined to be more likely to respond 
to virtual object 1 than the group 2 subscribers, then group 1 could be given a higher 
percentage than group 2. Table E also shows that virtual object 1 is not applicable to groups 
5 and 6 because only four groups are defined for the target category ADI Thus, all the 
reception sites will be grouped into one of groups 1 through 4. 

Table E - Virtual Object Ranking Percentages 



Virtual Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Virtual object 1 


1 


50 


50 


0 


0 


N/A 


N/A 




2 


30 


10 


20 


40 


N/A 


N/A 


» 


3 


0 


0 


0 


0 


0 


0 


Virtual object 2 


1 


0 


0 


50 


50 


N/A 


N/A 




2 


0 


0 


0 


0 


N/A 


N/A 




3 


0 


0 


0 


0 


0 


0 


Virtual obiect3 


1 


0 


0 


0 


0 


N/A 


N/A 




2 


25 


25 


25 


25 


N/A 


N/A 




3 


0 


o 


0 


0 


o 


0 
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1 

2 
3 
4 
5 
6 
7 
8 
9 

10 
11 
12 
13 
14 
15 



Virtual Object 


i argei 


Group 


Group 


Group 


Group 


Group 


f-livtiin 

VXl UU|J 




Category 


i 


L 


-a 


A 




O 


Virtual obiect 4 


1 


50 


0 


50 


0 


XT/ A 

N/A 


XT/ A 

N/A 




2 


0 


40 


40 


20 


XT y A 

N/A 


XT/ A 

N/A 




3 


10 


30 


10 


30 


10 


10 


Virtual obiect 5 


1 


40 


20 


20 


40 


XT/ A 

N/A 


XT/ A 

N/A 




2 


10 


30 


30 


30 


XT/ A 

N/A 


XT/ A 

JN/A 




3 


30 


30 


30 


5 


5 


0 


Virtual obiect 6 


1 


0 


0 


0 


0 


XT/ A 

N/A 


XT/ A 

N/A 




2 


0 


0 


0 


0 


XT/ A 

N/A 


XT/ A 

N/A 




3 


10 


1U 


1 f\ 

1U 


1U 


oO 




Virtual obiect 7 


1 


20 


40 


40 


20 


N/A 


N/A 


a 


2 


25 


25 


25 


25 


N/A 


N/A 


a 


3 


0 


30 


20 


30 


0 


20 


Virtual obiect 8 


1 


30 


40 


0 


30 


N/A 


N/A 


a 


2 


30 


30 


10 


30 


N/A 


N/A 


a 


3 


20 


o 


20 


20 


20 


20 



1 6 Using this paradigm, virtual objects can be targeted using at least two methods. The 

17 first is a designated multi-virtual object campaign where specific unique sets of groups are 

1 8 assigned for each virtual object 3 8 of the campaign. In the second method, each virtual object 

19 38 provided by an advertiser is independently associated with groups. Virtual objects from 

20 several different advertisers are then used together to optimize use of virtual object locations. 

2 1 As depicted in Figure 14, blocks 2368, 2370, 2372, and 2374, the virtual object placement 

22 engine 1 307 determines: 1) how many virtual objects are assigned to which virtual object 

23 location; 2) which target category is used for which virtual object location; 3) which virtual 

24 objects to place in each virtual objects location; and 4) which groups are assigned to which 

25 virtual objects, respectively. To limit the need for excessive distribution bandwidth to distribute 

26 virtual objects to reception sites, the algorithm in the virtual object placement engine 1307 that 

27 assigns targeted virtual objects to the virtual objects assumes that there is a total number of 

28 virtual objects available [TOTAL_VIRTUAL OBJECTS] for a segment of content 36 (across 

29 all virtual object locations), and assumes that no more than some maximum number of the 

30 virtual objects can be or are desired to be assigned to a given virtual object location 37. This 

3 1 amount is denoted as [MAXVIRTUAL OBJECTS]. 

32 Figure 15 presents an embodiment of a process used by the virtual object placement 

33 engine 1 307 to execute the functions listed in blocks 2368, 2370, 2372, and 2374 depicted 
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1 in Figure 14. The process begins with the start ellipse, 23 1 8. In block 2320, the virtual object 

2 placement engine 1307 determines the virtual object 38 best suited for each virtual object 

3 location 37 for all target categories. In block 2322, the virtual object placement engine 1 307 

4 determines the best virtual object/target category combination for each virtual object location 

5 37. Inblock 2324, the virtual object placement engine 1307 compares virtual object/target 

6 category combinations for all virtual object locations. In block 2326, the virtual object 

7 placement engine 1 307, for a virtual object location 37 and target category, determines the best 

8 virtual objects to associate with the virtual object location 37. In block 2328, the virtual object 

9 placement engine 1 307 repeats block 2326 for each target category. In block 2330, the virtual 

10 object placement engine 1307 determines the target category that yields the optimum placement 

1 1 of virtual objects for a virtual object location 37. Inblock2332, the virtual object placement 

12 engine 1307 repeats blocks 2326, 2328, and 2330 for all virtual object locations. Inblock 

1 3 2334, the virtual object placement engine 1 307 determines the best combination of multiple 

1 4 virtual objects for each virtual object location 37. In block 2336, for the remaining virtual 

1 5 object locations, the virtual object placement engine 1 307 assigns the best matching virtual 

1 6 object 38. The process ends with block 2338. 

17 A further embodiment of a virtual objects targeting algorithm presented in Figure 15 

1 8 will be described with reference to the example values shown in Tables A-E. Various other 

19 prioritizing or ranking schemes may be used as described later. 

20 Step 1: In block 2320 in Figure 15, the virtual object placement engine 1307, fora 

2 1 virtual object location 37, determines the virtual objects with the highest overall ranking if that 

22 virtual object 38 were the only virtual object 38 to be placed in a virtual object location 37 in 

23 the content 36. This step compares the data in Tables D and E. Figure 1 6 and the description 

24 that follows below present a more detailed embodiment of several of the blocks presented in 

25 Figure 15. In step la, as an embodiment of block 2421 in Figure 16, the virtual object 

26 placement engine 1 307 selects the first virtual object location 37 and as an embodiment of 

27 block 242 1 in Figure 1 6, selects the first virtual object 3 8 to be analyzed. As Step lb, for that 

28 virtual object selected in Step la, the virtual object placement engine 1307 selects the first 

-36- Attorney Docket 5219 



1 category, as an embodiment of block 2423 in Figure 16. Then, the virtual object placement 

2 engine 1 307 multiplies the virtual object's Group Ranking Percentage by the virtual object 

3 location's Group Breakdown Percentage for each group as an embodiment of block 2424 in 

4 Figure 16 and sums the result, as an embodiment ofblock 2425 in Figure 16. AsStep lc,the 

5 virtual object placement engine 1307 repeats Step lb for the next target category, as an 

6 embodiment of block 2426 in Figure 16. As Step Id, the virtual object placement engine 1307 

7 repeatssteps lband lcforeach virtual object 38, as an embodiment ofblock 2427 in Figure 

8 16. As Step le, for the virtual object location 37 under consideration, the virtual object 

9 placement engine 1307 selects the virtual object/target category that yields the highest summed 

1 0 value, as an embodiment of block 2428 in Figure 1 6. Then, for Step 1 f , the virtual object 

11 placement engine 1307 repeats Steps lb-leforallvirtualobjectlocations,asanembodiment 

12 of block 2429 in Figure 16. 

13 For example, using virtual object location 1, virtual object 1: 

14 target category 1: 50*25 + 50*25 + 0*25 + 0*25 = 25% 

15 target category 2: 30*30 + 10*10 + 20*20 + 40*40= 30% 

16 target category 3: 0*10 + 0*10 + 0*20 + 0*20 0*20 + 0*20= 0% 

17 The cross-multiplied result then shows a measure of effectiveness for each virtual 

18 object 38 if displayed in the corresponding virtual object location 37. Table F below presents 

19 the results of Step 1 above for virtual object location 1. 

20 Table F 



26 



27 



Virtual 
object 
location 
/ Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 


1/1 


1 


12.5 


12.5 


0 


0 


0 


0 


25 




2 


9 


1 


4 


16 


0 


0 


30 




3 


0 


0 


0 


0 


0 


0 


0 


1/2 


1 


0 


0 


12.5 


12.5 


0 


0 


25 




2 


0 


0 


0 


0 


0 


0 


0 
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Virtual 

object 

location 

/Virtual 

object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 




o 
D 


A 
U 


n 
u 


n 
u 


n 

u 




0 


0 


1/3 


1 
1 


fi 


n 

u 


n 
u 


n 
\j 


n 


o 


o 




I 


f.J 


z.o 


z 

J 




n 
u 


n 

V/ 


OS 




3 


(J 


0 


A 

0 


A 

u 


u 


n 
u 


n 


1/4 


1 


12.5 


r\ 
U 


1Z.J 


A 

u 


A 
U 


A 

u 






2 


0 


4 




o 
o 


A 

u 


A 

u 


zu 




3 


1 


3 


Z 


o 


z 


L 


ID 


1/5 


1 


10 


5 


< 


f 
D 


A 

u 


A 
U 


OS 




2 


3 


3 


0 


1Z 


A 

u 


n 
u 


OA 




3 


3 


3 


0 


1 
1 


1 
I 


u 


\A 


1/6 


1 


0 


0 


A 

0 


A 


A 

u 


A 

u 


n 
U 




2 


0 


o 


A 
0 


A 

u 


A 

u 


A 

u 


n 

V 




3 


1 


1 

1 


Z 


Z 


0 


o 


1 8 

10 


1/7 


1 


3 


c 

J 


1 A 


c 


A 

u 


A 






2 


f .D 


Z.J 


c 

J 


1 n 

IU 


n 
u 


n 


OS 






n 
U 


a 
j> 




yj 


o 


4 


17 


1/8 


1 


7.5 


10 


0 


7.5 


0 


0 


25 




2 


9 


3 


2 


12 


0 


0 


26 




3 


2 


0 


4 


4 


4 


4 


18 



7 Step 2: Returning to Figure 15 ,foreach virtual objectlocation37, the virtual object 

8 placement engine 1307, in block 2322, determines the virtual object/target category 

9 combination that results in the highest overall ranking. In one embodiment the virtual object 

1 0 placement engine 1 307, lists the virtual object locations, the overall ranking, the corresponding 

1 1 virtual object 38, and the corresponding target category. In case of a tie, the virtual object 

12 placement engine 1307 selects anyvirtual object 38 with the overall highest ranking. TableG 
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1 shows the results. Thus, from Table G, virtual object 4, a virtual object 3 8 displayed within 

2 virtual object location 4 yields a measure of effectiveness of 50 (highest) and virtual object 8 

3 along within virtual object location 5 yields a measure of effectiveness of 28. 

4 Table G 



5 


Virtual Object Location 


Highest Overall 
Ranking 


Corresponding 
Virtual Object 


Corresponding Target 
Category 


6 


Virtual object location 1 


30 


Virtual Object 1 


2 


7 


Virtual object location 2 


35 


Virtual Object 2 


1 


8 


Virtual object location 3 


35 


Virtual Object 1 


1 


9 


Virtual object location 4 


50 


Virtual Object 4 


1 


10 


Virtual object location 5 


28 


Virtual Object 8 


2 



11 



12 Step3: In one embodiment of block 2324 in Figure 15, the virtual object placement 

1 3 engine 1 307 orders the resulting list of virtual object locations from Step 2 from lowest overall 

14 ranking to highest overall ranking to compare virtual object/target category combinations for 

15 virtual object locations. Table H shows the results. 



16 Table H 



17 


Virtual Object Location 


Overall Ranking 


Corresponding 
Virtual Object 


Corresponding Target 
Category 


18 


Virtual object location 5 


28 


Virtual Object 8 


2 


19 


Virtual object location 1 


30 


Virtual Object 1 


2 


20 


Virtual object location 2 


35 


Virtual Object 2 


1 


21 


Virtual object location 3 


35 


Virtual Object 1 


1 


22 


Virtual object location 4 


50 


Virtual Object 4 


1 



23 Step 4: hi one embodiment of block 2326 in Figure 15, the virtual object placement 

24 engine 1 307 uses the process shown in Figure 17 to determine the best virtual objects to 

25 associatewithavirtualobjectlocation37. The blockbegins with ellipse 2440. Inblock2441 

26 in Figure 17, the virtual object placement engine 1 307 selects the virtual object location 37 
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1 from Step 3 resulting in the lowest overall ranking. As Step 4a, for the selected virtual object 

2 location 37, the virtual object placement engine 1 307 selects the first target category, as an 

3 embodiment of block 2442 in Figure 17. As Step 4b, the virtual object placement engine 1 307 

4 assembles a table showing the product of each virtual object Group Ranking Percentage and 

5 virtual object location Group Breakdown Percentage combination. Table I below provides 

6 an example for virtual object location 5 and target category 1. 



7 Table I 



8 
9 
10 


Virtual Object 
Location / Virtual 
Object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Summation 


11 


5/1 


1 


10 


15 


0 


0 


25 


12 


5/2 


1 


0 


0 


15 


10 


25 


13 


5/3 


1 


0 


0 


0 


0 


0 


14 


5/4 


1 


10 


0 


15 


0 


25 


15 


5/5 


1 


8 


6 


6 


4 


24 


16 


5/6 


1 


0 


0 


0 


0 


0 


17 


5/7 


1 


4 


6 


12 


4 


26 


18 


5/8 


1 


6 


12 


0 


6 


24 



19 As Step 4c, as an embodiment of block 2443 in Figure 17, the virtual object 

20 placement engine 1 307 finds the product that is the highest. In case of a tie, the virtual object 

21 placement engine 1307 selects the product that corresponds to the highest summation value for 

22 that virtual object location / virtual object combination. In case a tie still persists, the virtual 

23 object placement engine 1 307 selects any of the cells with an equivalent value. Table J below 

24 shows the previous example continued where group 2 for virtual object location / virtual object 

25 combination 5/1 is selected. 
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1 



Table J 



3 
4 


Virtual Object 
Location / Virtual 
Object 


1 arget 
Category 


LrrOUp 

1 


Group 
2 


IjTOUp 

3 


Group 
4 


Summation 


5 


5/1 


1 


10 


*15* 


0 


0 


25 


6 


5/2 


1 


0 


0 


15 


10 


25 


7 


5/3 


1 


0 


0 


0 


0 


0 


8 


5/4 




10 


0 


15 


0 


25 


9 


5/5 




8 


6 


6 


4 


24 


10 


5/6 




0 


0 


0 


0 


0 


11 


5/7 




4 


6 


12 


4 


26 


12 


5/8 




6 


12 


0 


6 


24 



13 



14 Step5: As an embodiment ofblock 2444 in Figure 17, the virtual object placement 

1 5 engine 1 307 finds the product that is next highest (or the same value as in Step 4), but that is 

1 6 associated with a group not yet selected. Again, in case of a tie, the virtual object placement 

1 7 engine 1 307 selects the product that corresponds to the highest summation value for that virtual 

1 8 object location / virtual object combination. In case a tie still persists, the virtual object 

1 9 placement engine 1 307 selects any of the cells with an equivalent value. Table K below shows 

20 the previous example continued. 



21 Table K 



22 
23 
24 


Virtual Object 
Location /Virtual 
Object 


Target 
Category 


Group 1 


Group 2 


Group 3 


Group 4 


25 


5/1 




*10* 


*15* 


0 


0 


26 


5/2 




0 


0 


*15* 


no* 


27 


5/3 




0 


0 


0 


0 


28 


5/4 




10 


0 


15 


0 


29 


5/5 




8 


6 


6 


4 


30 


5/6 




0 


0 


0 


0 
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v inudi wujcci. 
Location /Virtual 
Object 


i <trgci 
Category 


vrrOUp 1 


Vjroup ^ 


vrrOlip J 


vjrrOUp *♦ 


1 


5/7 


1 


4 


6 


12 


4 


2 


5/8 


1 


6 


12 


0 


6 



3 Step 6: As an embodiment of block 2446 in Figure 17, the virtual object placement 

4 engine 1 307 repeats Step 5 until a product has been selected for all groups. Table L below 

5 continues the example. 

6 Table L 



7 
8 
9 


Virtual Object 

Location / 
Virtual Object 


Target 
Category 


Group 1 


Group 2 


Group 3 


Group 4 


10 


5/1 




*10* 


*15* 


0 


0 


11 


5/2 




0 


0 


*15* 


*10* 


12 


5/3 




0 


0 


0 


0 


13 


5/4 




10 


0 


15 


0 


14 


5/5 




8 


6 


6 


4 


15 


5/6 




0 


0 


0 


0 


16 


5/7 




4 


6 


12 


4 


17 


5/8 




6 


12 


0 


6 



18 Step 7: As an embodiment of block 2448 in Figure 17, for all virtual objects with 

19 products cells selected in Step 6, the virtual object placement engine 1307 calculates the 

20 summed products of those selected cells for each virtual object 38 . Table M below shows the 

21 results. 



22 Table M 



23 


Virtual Object 


Target 


Group 1 


Group 2 


Group 3 


Group 4 


Summation 


24 


Location / 


Category 












25 


Virtual Object 














26 


5/1 


1 


*10* 


*15* 


0 


0 


25 
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Virtual {"^fiiA^f 
V irlUd.1 VsUJcH 

Location / 
Virtual Object 


1 41 gel 

Category 


vrruup J. 


\jxuup m 


vrruup o 


vrruup *r 


OUIIlIilallllll 


1 

1 


5/2 


1 


0 


0 


*15* 


*10* 


25 


2 


5/3 


1 


0 


0 


0 


0 


0 


>-> 
3 


5/4 


1 


10 


0 


15 


0 


0 


4 


5/5 


1 


8 


6 


6 


4 


0 


5 


5/6 


1 


0 


0 


0 


0 


0 


6 


5/7 


1 


4 


6 


12 


4 


0 


7 


5/8 


1 


6 


12 


0 


6 


0 



8 Step 8: As an embodiment of block 2450 in Figure 17, the virtual object placement 

9 engine 1 307 orders the virtual objects in Step 7 from highest summed value to lowest. In case 

10 of equal summed values, the virtual object placement engine 1307 arbitrarily orders those 

1 1 virtual objects with the same summed value. Table N presents the example results. 

12 Table N 



Virtual Object 


Target 


Group 1 


Group 2 


Group 3 


Group 4 


Summation 


Location / 


Category 












Virtual Object 














5/1 


1 


10 


15 


0 


0 


25 


5/2 


1 


1 


0 


15 


10 


25 



18 Step 9: As Step 9a, if the number of virtual objects selected in Step 8 exceeds 

19 [MAXJVIRTUAL OBJECTS], the virtual object placement engine 1307 selects the first 

20 [MAX_VIRTUAL OBJECTS] virtual objects with the summed value as an embodiment of 

21 block 2452 in Figure 17. For example, if it is desired to assign at most two virtual objects to 

22 a virtual obj ect location 37 , the virtual object placement engine 1 307 selects the two virtual 

23 objects with the highest virtual object Group Ranking Percentage and virtual object location 

24 Group Breakdown Percentage products. Next, as Step 9b, for the unselected virtual objects, 
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1 the virtual object placement engine 1 307 determines those groups that were associated with 

2 these omitted virtual objects, as an embodiment of block 2454 in Figure 17. 

3 Step 10: As an embodiment of block 2456 in Figure 17, for the virtual objects 

4 associated with the groups determined in Step 9b, the virtual object placement engine 1 307 

5 selects the product within that group that is the highest for the [MAXJVTRTUAL OBJECT] 

6 selected virtual objects from Step 9a. The virtual object placement engine 1 307 recalculates 

7 the summed products of those selected groups cells for each of the virtual objects. Table O 

8 below provides a new example, assuming [MAX_VIRTUAL OBJECTS] = 2; therefore, 

9 groups 5 and 6, which are associated with virtual object 6, may be reallocated to virtual objects 
10 7 & 5, respectively. 



11 Table O 

12 Result before Step 10 is shown below: 



13 
14 
15 
16 
17 


Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 


18 


5/7 


3 


0 


#9* 


2 




0 


2 


18 


19 


5/5 


3 


*3* 


9 


*3* 


1.5 


0.5 


0 


6 


20 


5/6 


3 


1 


3 


1 


3 


#3* 


#3* 


6 


21 


Result after Step 10 is shown below: 














22 
23 
24 
25 
26 


Virtual 
object 
location / 

Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 


27 


5/7 


3 


0 


#9* 


2 


*9* 


0 


*2* 


20 


28 


5/5 


3 


*3* 


9 




1.5 


*0.5* 


0 


6.5 


29 


5/6 


3 


1 


3 


1 


3 


3 


3 


0 



-44- 



Attorney Docket 5219 



1 Step 1 1 : As an embodiment of block 2458 in Figure 17, the virtual object placement 

2 engine 1 307 calculates the total summed product value for all virtual objects selected in Step 

3 10. From Table P, this value is 26.5. The resultant groups selected for each virtual object 38 

4 will serve as the group assignments if this virtual object location / target category ultimately 

5 results in the best match, as determined in the remaining steps of the algorithm. 



6 Table P 



7 


Virtual 


Target 


Group 


Group 


Group 


Group 


Group 


Group 


Summation 


8 


object 


Category 


1 


2 


3 


4 


5 


6 




9 


location / 


















10 


Virtual 


















11 


object 


















12 
13 
14 


5/7 


3 


0 


*9* 


2 


*9* 


0 


*2* 


20 


5/5 


3 


*3* 


9 




1.5 


0.5 


0 


6.5 


Total summed product values 


26.5 



1 5 Step 1 2: The virtual object placement engine 1 307 repeats steps 4-11 above for the 

1 6 same selected virtual object location 37 of Step 4 using the remaining target categories, as an 

17 embodiment ofblock 2328 in Figure 15. The Table Q example below provides the output 

18 results for each of the three example target categories. 



19 Table Q 



20 
21 
22 
23 
24 


Virtual 
object 
location / 

Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Summation 


25 


5/1 


1 


*10* 


*15* 


0 


0 


25 


26 


5/2 


1 


0 


0 


*15* 


*10* 


25 


27 


Total summed product values 


50 



-45- 



Attorney Docket 5219 



1 

2 
3 
4 

C 
J 


Virtual 
object 
location / 
Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 


6 


5/1 


2 


#9* 


3 


2 


*12* 


0 


0 


21 


7 


5/4 


2 


0 




#4* 


6 


0 


0 


16 


8 


Total summed product values 


37 



9 
10 
11 
12 
13 


Virtual 
object 
location 
/Virtual 
object 


Target 
Category 


Group 
1 


Group 
2 


Group 
3 


Group 
4 


Group 
5 


Group 
6 


Summation 


14 


5/7 


3 


0 


*9* 


2 




0 


*2* 


20 


15 


5/5 


3 


*3* 


9 


*3* 


1.5 


*0.5* 


0 


6.5 


16 


Total summed product values 


26.5 



17 Step 13: As an embodiment ofblock 2330 in Figure 15, the virtual object placement 

1 8 engine 1 307 selects the target category that yields the highest total summed product amount. 

1 9 The virtual object placement engine 1 307 assigns this as the Maximum Rank for that virtual 

20 object location 37. In the case above, the virtual object placement engine 1 307 would assign 

21 target category 1, with a value of 50 that is selected. 

22 Step 14: As an embodiment ofblock 2332 in Figure 15, the virtual object placement 

23 engine 1 307 repeats Steps 4-13 for the virtual object location 37 selected in Step 4 with the 

24 next lowest overall ranking, computing the Maximum Rank for each virtual object location 37. 

25 Step 15: As an embodiment ofblock 2334 in Figure 15, the virtual object placement 

26 engine 1307 uses the available [MAX_VIRTUAL OBJECTS] virtual objects for the virtual 

27 object locations up to the maximum number of [TOTAL_VIRTUAL OBJECTS] thatyieldthe 

28 largest Maximum Rank. The virtual object placement engine 1 307 makes use of the relevant 

29 target category determined in Step 13, with virtual objects as determined in Step 10, with 

30 group assignments as determined in Step 1 1 . 
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1 Step 16: As an embodiment ofblock 2336 in Figure 15, for all other virtual object 

2 locations, the virtual object placement engine 1307 assigns the single virtual objects that yielded 

3 the highest Overall Ranking as determined in Step 2. 

4 The above algorithm performed by the virtual object placement engine 1307ismeant 

5 to be illustrative and not limiting. Other algorithms are possible for assigning targeted 

6 advertising to groups of reception sites or to individual reception sites. Other targeted 

7 advertising routines can also be used by the virtual object placement engine 1307. 

8 The above algorithm can be simplified in the case where virtual obj ects are being 

9 selected to be delivered with the content 36 to be received by a single subscriber or reception 

10 site 30. In this case, prior to initiating the steps in the algorithm, the virtual object location 

1 1 Group Breakdown Percentages table may be modified to display a group breakdown 

1 2 percentage of 0 for all groups that the subscriber does not belong to for each target category. 

13 An alternate virtual object targeting routine 1374 is described in U.S. Patent 

14 5,600,364, to Hendricks, John S, entitled NETWORK CONTROLLER FOR CABLE 

1 5 TELEVISION DELIVERY SYSTEM, which is hereby incorporated by reference, hi this 

1 6 alternative, software in the virtual object targeting system 1220 generates packages of virtual 

17 objects geared towards particular subscribers and makes use of a subscriber's demographic 

1 8 information and viewing habits to determine those virtual objects that are of most interest to that 

19 particular subscriber. The routine 1374 then outputs packages of virtual objects targeted 

20 towards each subscriber or group of subscribers. 

2 1 Figure 1 8 shows the seven primary functions of an alternate virtual object targeting 

22 routine 1 374. The function of the routine 1 374 is to target virtual objects for reception sites 

23 based on historical programs watched data and other data that is available at the TVOMS 300. 

24 In the discussion that follows, the alternate virtual object targeting routine 1 374 is described as 

25 executed at the TVOMS 300. 

26 Theprocessmaybeinitiatedasshownatinitiation ellipse 1420. In the first subroutine, 

27 identified at block 1422, the virtual object targeting system 1220 determines the programs 

28 watched matrices stored in the subscriber information database 1210. The determine programs 
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1 watched matrices subroutine 1 422 uses a unique reception site ID to access a specific matrix 

2 for one reception site. These matrices are maintained and updated by periodic collections by 

3 the operations center 10 of accumulated information from the reception sites. 

4 In the second subroutine, shown at block 1424, the virtual object targeting system 

5 1 220 develops other matrices based on other available information. To develop other matrices 

6 based on other available information subroutine 1424 is an optional subroutine not required for 

7 the functioning of the system. For groups of reception sites or for each individual reception site, 

8 matrices maybe developed based on the demographic information, billing information, pricing 

9 information, age information and other information that may be stored in the subscriber 

1 0 information database 1210. 

11 hi the process matrices through correlation algorithms subroutine, block 1426, the 

12 virtual object targeting system 1220 processes all matrices through a set of correlation 

1 3 algorithms. In particular, the virtual object targeting system 1220 takes matrices developed in 

14 the first two subroutines 1422 and 1424 and processes the matrices until reaching a final matrix. 

1 5 Figure 19 shows an embodiment of the matrices processing subroutine 1426 that is 

16 calledbythe virtual objects targeting sequence 1374 shown in Figure 18. As shown in Figure 

17 19, the virtual object targeting system 1220 initiates the matrices processing subroutine 1426 

18 at initiation ellipse 1427 and then accesses or queries, at block 1420, the programs watched 

1 9 file and gathers information regarding either an individual subscriber or a group of subscribers. 

20 The virtual object targeting system 1 220 can gather the programs watched information in this 

21 way for individual subscribers or a group of subscribers. 

22 Once the programs watched information has been gathered in the database, the virtual 

23 object targeting system 1220 selects and groups, at block 1430, programs watched categories 

24 and time periods. The software initially takes each program category (e.g., sports, news, 

25 mysteries, etc.) and determines the number of programs watched for a given time. The periods 

26 may be set to any length of time, including, for example, one, two, three or four weeks. The 

27 virtual object targeting system 1 220 will loop through such a counting process for each group 

28 and period and then proceed to build a programs watched matrix, at block 1432, based on the 
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1 program categories and periods. Essentially, all programs watched in a particular category and 

2 time period will be entered into the programs watched matrix. Once the matrix has been built, 

3 the virtual object targeting system 1220, using matrices processing subroutine 1426, will 

4 process the matrix for a given subscriber or group of subscribers through the correlation 

5 algorithms. 

6 A number of correlation algorithms may be used to weight each selected program 

7 category. For example, as shown at block 1434, the virtual object targeting system 1220 may 

8 use a sum of squares algorithm to determine the weighting. Once weighted, the weighted 

9 categories will be correlated by the virtual object targeting system 1220atblock 1436, with 

10 various virtual objects stored in the available virtual objects database 1265. The virtual object 

1 1 targeting system 1220 then selects a set of the most heavily weighted virtual objects for 

12 inclusion within the content 36 to be delivered to individual subscribers or groups of 

1 3 subscribers. Having determined the weightings of each group and prioritizing the groups 

14 accordingly, the virtual object targeting system 1220 returns, block 1438, to the virtual objects 

1 5 targeting sequence 1 374 of Figure 18. 

16 Referring back to Figure 18, in the fourth subroutine, as represented at block 1428, 

1 7 the virtual object targeting system 1 220 uses the final matrix developed by the correlation and 

1 8 weighing algorithm described above, to select a grouping (or selective filter) for each reception 

19 site 30. The final groupings of virtual objects that maybe sent to the reception sites or group 

20 of reception sites may use a subroutine as diagramed in Figure 20. 

2 1 The fourth subroutine 1428 , depicted in Figure 20, is called or initiated by the virtual 

22 objects targeting sequence 1374ofFigure 18 in order to determine the final groupings. In the 

23 subroutine shown at block 1444, the virtual object targeting system 1220 selects a set of virtual 

24 objects that will be used in the chosen groupings. This selection process may involve virtual 

25 objects from various virtual objects categories. Each virtual object 38 may subsequently be 

26 assigned a number of times that it will be shown in a given segment of content 36. The 

27 frequency of display may be based on various factors, including the number of requests and 

28 cost paid by the respective advertisers to have the virtual objects displayed, as shown in block 
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1 1 446. Such factors may be used by the virtual object targeting system 1 220 in the next step 

2 of the subroutine, at block 1448 , at which the virtual object targeting system 1 220 assigns a 

3 weighting to specific virtual objects in each virtual objects category. These weightings are used 

4 to prioritize the virtual objects that will be sent to individual reception sites or group of reception 

5 sites. 

6 Once the virtual objects have been weighted, the virtual object targeting system 1220 

7 executes a correlation algorithm, at block 1450, using selected criteria (i.e., the various factors 

8 used to weight the virtual objects) as well as the output of each programs watched matrix. Any 

9 number of correlation algorithms and weighting algorithms maybe used, including the sum of 

10 squares weighting algorithm described above. 

1 1 The results from the correlation algorithm subsequently determine the virtual objects 

1 2 and program content 36 that is sent to the virtual object targeting system 1 220 for distribution. 

1 3 Once the virtual object targeting system 1 220 at the fourth subroutine 1428 completes these 

14 steps, the subscriber information database 1210 updates the subscriber record based on the 

15 virtual objects that are sent, as shown at block 1454. The database update allows the 

1 6 advertisers to track the costs and frequency of the virtual objects targeted to specific reception 

17 sites or groups of reception sites. Following the updates, the virtual object targeting system 

18 1220 returns to the virtual objects targeting sequence shown in Figure 18, block 1456. 

1 9 Referring to Figure 2 1 , reception site groupings ( 1 through 5) 1460 are shown. The 

20 number of reception site groupings available may be determined by the bandwidth available to 

2 1 transmit virtual objects along with content 36. The available bandwidth or resources provided 

22 by the delivery network 1 1 may limit the number of virtual objects that are available to 

23 distribute to the reception site 30. 

24 Referring back to Figure 1 8, the virtual object targeting system 1 220 at the fifth 

25 subroutine, represented at block 1466, prepares reception site group information for 

26 transmission to the reception sites along with the requested content 36. 

27 In the sixth subroutine, block 1468, the virtual object targeting system 1220selects 

28 the targeted virtual objects. The sixth subroutine 1468 is the last decision making process in 
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1 displaying a targeted virtual objects for a subscriber. As shown in block 1469, the reception 

2 site 30 then displays the targeted virtual objects with the content 36. 

3 As noted above, targeted advertising can be based on viewing a specific program or 

4 a category of programming content 36. In an embodiment, the reception site 30 performs this 

5 last step by correlating (or matching) the program being watched by the subscriber with the 

6 reception site group information that has been previously transmitted by the T VOMS 300. 

7 Figure 21 shows an exemplary table matching reception site groups 1460 and program 

8 categories 1470 with specific virtual objects. The virtual objects are shown in Figure 22 at 

9 1474 and are assigned Roman numerals I through X, for example. The number of reception 

1 0 site groupings and virtual objects can vary. Figure 22 shows a division of available bandwidth 

1 1 to carry ten virtual objects. In this example, the virtual objects 1474 are numbered 

12 1101-1110. 

1 3 The TVOMS 300 will transmit group information to a reception site 30 shown as row 

14 names 1460 on Figure 21. The TVOMS 300 will also transmit data that informs the reception 

1 5 site 30 which of the multiple virtual objects 1474 is assigned to a program category shown as 

1 6 columns 1 470 on Figure 2 1 . Each reception site 30 only requires the data related to that 

1 7 reception site's assigned group (or row) . For example, in Figure 2 1 , the reception site 30 in 

1 8 group 1 (row 1) is provided with data on the virtual objects which are assigned for sports 

1 9 program as I, children's program as IV and mystery category program as IQ. In this manner, 

20 each reception site 30 is only required to store information related to its own grouping. 

2 1 Therefore, a reception site 30 that is in group 1 only needs to store the information related to 

22 group 1 that is found in row 1 of Figure 21. 

23 Figure 23 shows a software program flow 1490 that is an alternative to the virtual 

24 object targeting system 1220 targeting routine 1374, depicted in Figure 18. The alternative 

25 routine 1490 allows each reception site 30 to be individually targeted with specific virtual 

26 objects. Preferably, it is initiated automatically, as shown at block 1492, by the TVOMS 300 

27 upon receipt of a program request from a reception site, for example, for a pay per view 

28 program. Thus, once the TVOMS 300 receives program request information from a reception 
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1 site, the T VOMS 300 begins the process of selecting a package of virtual objects that may be 

2 based on, among other things, that subscriber's demographic information and viewing 

3 history. 

4 Upon receipt of a program request from a reception site, the virtual object targeting 

5 system 1220 reads the reception site identifier, as shown at block 1494, and the program 

6 requested. The subscriber data collection engine 1 202 writes information on the program 

7 requested to the subscriber information database 1210, updating the subscriber record that 

8 contains listings of all programs requested within the past week, month or year. 

9 With continued reference to Figure 23 , the virtual object targeting system 1 220 then 

1 0 calls a subroutine that sorts the programs requested by program category, block 1498. In turn, 

1 1 the program categories are sorted, as shown at block 1 500, based on the number of times that 

1 2 program appearing in each particular category is requested. In so doing, virtual object targeting 

13 system 1220, using the sorting subroutine as shown at block 1500, determines and ranks those 

14 programs and program categories that are most frequently viewed at that reception site. 

1 5 All rankings of programs and program categories for that reception site 30 are written 

16 to the subscriber information database 1210, as shown at block 1502. 

17 Next, the virtual object targeting system 1220 calls a subroutine, shown at block 

18 1 504, that correlates the updated subscriber record with the available virtual objects database 

19 1265. By correlating these two with one another, the subroutine assigns or correlates various 

20 categories of virtual objects to each ranking of programs and program categories. The 

2 1 categories of virtual objects that may be so assigned are found in the available virtual objects 

22 database 1 265 and may include: ( 1 ) Household Goods/Products, (2) Home Improvement and 

23 Maintenance, (3) Personal Hygiene, (4) Entertainment Items and Events, (5) Sporting Goods 

24 and Events, (6) Motor Vehicles and Related Products, (7) Foodstuffs and Beverages, and (8) 

25 Miscellaneous, for example. Where, for example, the subscriber has watched a sporting 

26 program, the Sporting Goods and Events, Home Improvement and Maintenance categories 

27 may be assigned to that particular sporting event/ program and Sports program category, for 

28 example. 
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1 Once the programs and program categories are correlated with the virtual objects 

2 categories in the available virtual objects database 1265, the virtual object targeting system 

3 1 220 calls a sorting subroutine 1 506 that ranks the correlated virtual objects categories based 

4 on other information in the database files. In one embodiment, this ranking is primarily based 

5 on data in the updated subscriber information database 1210, as shown at block 1506. By 

6 using data on the subscriber's past program selections and demographic information, the virtual 

7 obj ect targeting system 1 220 ranks the correlated categories of virtual objects according to 

8 those likely to be of most interest to that subscriber. 

9 After the virtual object categories have been sorted and ranked, the virtual object 

1 0 targeting system 1 220 selects the top three virtual objects categories as the targeted categories 

11 for a given program and subscriber, block 1508. Individual virtual objects are then chosen 

1 2 from the available virtual objects database 1 265, with all selections made from the targeted 

1 3 categories, at block 1510. The virtual objects that are selected are written to the subscriber 

14 information database 1210 and to the content and virtual object packager 30, from where 

15 packages can be generated, at block 1512, for ultimate delivery to the reception site. 

16 Figure 24 depicts the object delivery center 15. The object delivery center 15 

1 7 receives content 36, virtual objects, retrieval plans, and other information from the operations 

18 center 10 that is to be transmitted to reception sites. The communication processor 16inthe 

1 9 object delivery center 1 5 may determine the delivery network and communications methods 

20 appropriate for each item to be delivered, may combine items to be delivered to common 

21 destinations, may format the items for delivery, and provide the formatted items to the 

22 processing router 17. The processing router 17 may then route each item to the appropriate 

23 modular connector 700, for example modular connector 700', modular connector 700", or 

24 modular connector 700 w , depending on the required delivery network 1 1 and communication 

25 method. 

26 A number of embodiments of delivery network 1 1 are presented below. The 

27 embodiments presented below may use the object delivery center 1 5, which inserts the virtual 

28 objects into the signal for delivery over the delivery network 1 1 . The embodiments presented 
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1 below use a modular connector 700 in the reception site 30, that receives the delivered signal 

2 with virtual objects, extracts the virtual objects, and provides the virtual objects to the storage 

3 management processor 710. The modular connector 700 supports the receive functionality for 

4 each unique delivery network 1 1 communication method embodiment. 

5 Figure 25 presents embodiments associated with the delivery of virtual objects over 

6 acoaxial or fiber cable system2701 to a reception site 30. Virtual objects are provided to the 

7 delivery network 1 1 by the object delivery center 15 or directly by the operations center 10. 

8 The signal is delivered over the cable system 270 1 . The signal may provide for the delivery of 

9 virtual objects, content 36 containing virtual object locations, and reception site configuration 

1 0 and control information. The signal may also provide for virtual object viewing data and 

1 1 interactive virtual object requests from the reception site 30 to the local data collection center 

1 2 40, to the central data collection center 50, or to the interactive object service center 60 or the 

1 3 signal may be a means to provide access to the Internet or other public network through which 

14 virtual objects or content 36 are delivered (not shown). The cable system 2701 may be a 

1 5 coaxial cable network, totally fiber network, hybrid fiber coax network, fiber to the curb 

1 6 network, or any other cable distribution technology. The signal over the cable system may be 

17 generated by a cable modem, in which an external cable modem 2702 is used to receive the 

1 8 signal and provide the embedded virtual objects to the modular connector 700 in the reception 

1 9 site 30 for processing. Alternatively, the reception site 30 may contain an internal cable modem 

20 2705, which receives the signal and provides the virtual objects to the modular connector 700 

21 for processing. 

22 In another embodiment, the signal delivered over the cable system is a video signal. 

23 In one embodiment, the video signal is an analog video signal. In another embodiment, the 

24 video signal is a digital video signal. The reception site 30 may contain an internal cable 

25 receiver/tuner/demodulator 2706 to process the signal, and provide the embedded virtual 

26 objects to the modular connector 700. A set top terminal 2703, or other device capable of 

27 receiving a cable video signal, such as a cable ready TV, or PC with cable tuner (not shown), 

28 may process the video signal and deliver the video signal to the connector 700 in the reception 
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1 site 30, which extracts the embedded virtual objects. Alternately, the set top terminal 2703 , 

2 or other such device, may extract the embedded virtual objects from the video signal and 

3 provide the virtual objects to the modular connector 700 in the reception site 30. 

4 In another embodiment, virtual objects may be embedded within the audio signal, 

5 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to 

6 extract the virtual objects from the audio signal. In one embodiment, the audio signal is an 

7 analog audio signal. In another embodiment, the audio signal is a digital audio signal. 

8 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

9 data stream, requiring an appropriate spread spectrum receiver and modular connector 700 

10 in the reception site 30 to extract the virtual objects. In this embodiment, the spread spectrum 

1 1 signal is transmitted in the same bandwidth as the video or audio signal, but below the noise 

12 level. 

1 3 Figure 26 presents embodiments associated with the delivery of virtual objects over 

14 a wireless broadcast system 2801 toareceptionsite30. Virtual objects are provided to the 

1 5 delivery network 1 1 by the object delivery center 1 5 or directly by the operations center 1 0. 

1 6 The signal is delivered over the wireless broadcast system 280 1 . The signal may provide for 

17 the delivery of virtual objects, content 36 containing virtual object locations, and reception site 

1 8 configuration and control information. The signal may also provide for virtual object viewing 

1 9 data and interactive virtual object requests from the reception site 30 to the local data collection 

20 center 40, to the central data collection center 50, or to the interactive object service center 

21 60 or the signal may be a means to provide access to the Internet or other public network 

22 through which virtual objects or content 3 6 are delivered. The wireless broadcast system may 

23 be a microwave multipoint delivery system (MMDS), local multipoint distribution system 

24 (LMDS), Instructional Television Fixed Service (ITFS) system, or any other wireless data, 

25 video, or telephony broadcast system, including point-to-point and point-to-multipoint 

26 microwave broadcast systems like those provided by Teligent, Winstar digital wireless 

27 network, and ATT's wireless system. The signal over the wireless broadcast system may be 

28 generated by a wireless modem, in which an external wireless modem 2802 is used to receive 
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1 the signal and provide the embedded virtual objects to the modular connector 700 in the 

2 reception site 30 for processing. Alternatively, the reception site 30 may contain an internal 

3 wireless modem 2805, which receives the signal and provides the virtual objects to the modular 

4 connector 700 in the reception site 30 for processing. 

5 In another embodiment, the signal delivered over the wireless broadcast system is a 

6 video signal. In one embodiment, the video signal is an analog video signal. In another 

7 embodiment, the video signal is a digital video signal. The reception site 30 may contain an 

8 internal wireless receiver/tuner/demodulator 2806 to process the signal, and provide the 

9 embedded virtual objects to the modular connector 700. A wireless set-top terminal 2803, or 

1 0 other device capable of receiving a wireless video signal, such as a TV, or PC with a wireless 

1 1 receiver and tuner, may process the video signal and deliver the video signal to the modular 

12 connector 700 in the reception site 30, which extracts the embedded virtual objects. 

1 3 Alternately, the set top terminal 2803, or other such device, may extract the embedded virtual 

14 objects from the video signal and provide the data to the modular connector 700 in the 

15 reception site 30. 

16 hi another embodiment, virtual obj ects may be embedded within the audio signal, 

17 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to 

1 8 extract the virtual objects from the audio signal. In one embodiment, the audio signal is an 

19 analog audio signal. In another embodiment, the audio signal is a digital audio signal. 

20 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

2 1 data stream, requiring an appropriate spread spectrum receiver modular connector 700 in the 

22 reception site 30 to extract the virtual objects. In this embodiment, the spread spectrum 

23 signal is transmitted in the same bandwidth as the video or audio signal, but below the noise 

24 level. 

25 Figure 27 presents embodiments associated with the delivery of virtual objects over 

26 a satellite broadcast system 2901 to a reception site 30. Virtual obj ects are provided to the 

27 delivery network 11 by the object delivery center 15 or directly by the operations center 10. 

28 The signal is delivered over the satellite broadcast system 2901 . The signal may provide for 
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1 the delivery of virtual objects, content 36 containing virtual object locations, andreception site 

2 configuration and control information. The signal may also provide for virtual object viewing 

3 data and interactive virtual object requests from the reception site 30 to the local data collection 

4 center 40, to the central data collection center 50, or to the interactive object service center 

5 60 or the signal may be a means to provide access to the Internet or other public network 

6 through which virtual objects or content 36 are delivered. The satellite broadcast system2901 

7 can be a direct broadcast system like DirecTV and EchoStar, a direct to home satellite 

8 broadcast system, video network distribution broadcast system, a point-to-point or 

9 point-to-multipoint data VSAT system, a digital audio broadcast system like WorldSpace, CD 

10 Radio, or XM, or a mobile data and telephony satellite broadcast system like Iridium, 

1 1 Teledesic, or Globalstar. Alternatively, the satellite broadcast system can be regionalized 

1 2 broadcast services or store and forward communication services hosted on high flying balloons 

13 or on airplanes that provide communication repeater services to an small geographic region. 

14 The signal over the satellite broadcast systemmay be generated by a satellite data modem, in 

15 which an external satellite data receiver 2902 is used to receive the signal and provide the 

16 embedded virtual objects to the reception site 30 modular connector 700 for processing. 

17 Alternatively, the reception site 30 may contain an internal satellite receiver 2905, which 

18 receives the signal and provides the virtual objects to the modular connector 700 in the 

19 reception site 30 for processing. 

20 In another embodiment, the signal delivered over the satellite broadcast system is a 

21 video signal. In one embodiment, the video signal is an analog video signal. In another 

22 embodiment, the video signal is a digital video signal. The reception site 30 may contain an 

23 internal satellite video receiver 2906 to process the signal, and provide the embedded virtual 

24 objects to the modular connector 700. A satellite receiver 2903, or other device capable of 

25 receiving a satellite video signal, such as aTV, or PC with satellite receiver, may process the 

26 video signal and deliver the video signal to the modular connector 700 in the reception site 30, 

27 which extracts the embedded virtual objects. Alternately, the satellite receiver 2903, or other 
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1 such device, may extract the embedded virtual objects from the video signal and provide the 

2 data to the modular connector in the reception site 258. 

3 In another embodiment, virtual objects may be embedded within the audio signal, 

4 requiring an appropriate audio-capable modular connector 700 in the reception site 30 to 

5 extract the virtual objects from the audio signal. In one embodiment, the audio signal is an 

6 analog audio signal. In another embodiment, the audio signal is a digital audio signal. 

7 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

8 data stream, requiring an appropriate spread spectrum receiver modular connector 700 in the 

9 reception site 30 to extract the virtual objects. In this embodiment, the spread spectrum 

1 0 signal is transmitted in the same bandwidth as the video or audio signal, but below the noise 

11 level. 

1 2 Figure 28 presents embodiments associated with the delivery of virtual objects over 

1 3 a wired data network 300 1 to a reception site 30. Virtual objects are provided to the deli very 

14 network 1 1 by the object delivery center 15 or directly by the operations center 10. The signal 

15 is delivered over the wired data network 300 1 . The signal may provide for the delivery of 

1 6 virtual objects, content 36 containing virtual object locations, and reception site configuration 

17 and control information. The signal may also provide for virtual object viewing data and 

1 8 interactive virtual object requests from the reception site 30 to the local data collection center 

1 9 40, to the central data collection center 50, or to the interactive obj ect service center 60 or the 

20 signal may be a means to provide access to the Internet or other public network through which 

21 virtual objects orcontent 36 are delivered. The wired data network 3001 can be metallic wire 

22 or fiber, supporting any of a number of communication standards including HDSL, ADSL, 

23 DSL, ISDN, T 1 , T3 , SONET, ATM, X.25, frame relay, Switched MultiMegabit Data Service 

24 (SMDS), or others. The signal sent over the wired data network may be generated by a data 

25 modem or transmission device, in which the appropriate modem, interface device, or Data 

26 Terminating Equipment (DTE) device is used to receive the signal and provide the embedded 

27 virtual objects to the reception site 30 modular connector 700 for processing. Embodiments 

28 of such receiving devices are shown in Figure 28 as HDSL modem 3002, ADSL modem 
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1 3003, DSL modem 3003, ISDN Terminal equipment (TE) device 3005, Tl Digital service unit 

2 (DSU) 3006, T3 DSU 3007, Fiber user network interface device (UNI) 3008, ATM UNI 

3 3009, X.25 DTE 3010, Frame relay assembler/disassembler (FRAD) 301 1, and SMDS 

4 subscriber network interface device (SNI) 3012. Alternatively, the reception site 30 may 

5 contain an internal modem or DTE 301 3, which receives one or more signal types andprovides 

6 the received signal with embedded virtual objects to the modular connector 700 in the 

7 reception site 30 for processing. Finally, the reception site 30 may be attached to a wired 

8 LAN using a transceiver. In this embodiment, virtual objects may be delivered over the LAN 

9 at any time. 

1 0 Figure 29 presents embodiments associated with the delivery of virtual objects using 

1 1 the public switched telephony network (PSTN) 3 10 1 to a reception site 30. Virtual objects 

12 are provided to the delivery network 1 1 by the object delivery center 1 5 or directly by the 

13 operations center 10. The signal is delivered over the PSTN 3 101. The signal may provide 

14 for the delivery of virtual objects, content 36 containing virtual object locations, and reception 

1 5 site configuration and control information. The signal may also provide for virtual object 

1 6 viewing data and interactive virtual object requests from the reception site 30 to the local data 

1 7 collection center 40, to the central data collection center 50, or to the interactive object service 

18 center 60 or the signal may be a means to provide access to the Internet or other public 

1 9 network through which virtual objects or content 36 are delivered. The signal sent over the 

20 PSTN may be generated by a data modem or transmission device, in which the appropriate 

2 1 modem 3 102 is used to receive the signal and provide the embedded virtual objects to the 

22 modular connector 700 in the reception site 30 for processing. Alternatively, the reception site 

23 30 may contain an internal modem 3 103, which receives the signal andprovides the received 

24 signal with embedded virtual objects to the modular connector 700 in the reception site 30 for 

25 processing. 

26 Figure 30 presents embodiments associated with the delivery of virtual objects using 

27 wirelesspersonalcommunicationssystem(PCS)3201 to a reception site 30. Virtual objects 

28 are provided to the delivery network 1 1 by the object delivery center 1 5 or directly by the 
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1 operations center 10. The signal is then delivered over the PCS network 320 L The wireless 

2 PCS systemmaybe,forexampleawirelessLAN, digital cellular telephony network, analog 

3 cellular telephony network, digital cellular radio system, analog cellular radio system, digital 

4 pager network, analog pager network, or Personal Communication Network (PCN). The 

5 signal may provide for the delivery of virtual objects, content 36 containing virtual object 

6 locations, and reception site configuration and control information. The signal may also provide 

7 for virtual object viewing data and interactive virtual object requests from the reception site 30 

8 to the local data collection center 40, to the central data collection center 50, or to the 

9 interactive object service center 60 or the signal may be a means to provide access to the 

1 0 Internet or other public network through which virtual objects or content 36 are delivered. A 

1 1 wireless PCS receiver 3202 is used to receive the signal and provide the embedded virtual 

1 2 obj ects to the modular connector 700 in the reception site 30 for processing. Alternatively, the 

1 3 reception site 258 may contain an internal wireless PCS receiver 3203, which receives the 

1 4 signal and provides the received signal with embedded virtual objects to the modular connector 

15 700 in the reception site 30 for processing. 

1 6 Figure 3 1 depicts several embodiments associated with the delivery of virtual objects 

1 7 using a national or local television broadcaster's signal. Virtual obj ects are provided to the 

18 either the national broadcaster 1110, the broadcast affiliate 1112, or the local cable system 

19 1114bytheobjectdeliverycenter 15 or directly by the operations center 10. Thesignalfrom 

20 the national broadcaster 1 1 10 can be delivered to reception site 30', 30" or 30'" using a 

21 satellite system 11 22, using a broadcast affiliate 1112terrestrially,orusingalocalcablesystem 

22 11 14. Alternatively, the local television broadcast affiliate 1 1 12 can originate the signal which 

23 can be delivered to the reception site 30', 30"or 30 w terrestrially, or using a local cable system 

24 1 1 14. The signal may provide for the delivery of virtual objects, content 36 containing virtual 

25 object locations, and reception site configuration and control information. The signal may also 

26 provide for virtual object viewing data and interactive virtual object requests from the reception 

27 sites 30', 30", and 30'" to the local data collection center 40, to the central data collection 

28 center 50, or to the interactive object service center 60 or the signal may be a means to provide 
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1 access to the Internet or other public network through which virtual objects or content 36 are 

2 delivered. In one embodiment, the video signal is an analog video signal and the virtual objects 

3 is embedded in the video signal. In another embodiment, the video signal is a digital video 

4 signal and the virtual objects are carried as an independent data stream. In another 

5 embodiment, virtual objects maybe embedded within the audio signal. In one embodiment, 

6 the audio signal is an analog audio signal. In another embodiment, the audio signal is adigital 

7 audio signal. 

8 In yet another embodiment, the signal is a spread spectrum signal containing a digital 

9 data stream, requiring an appropriate spread spectrum receiver modular connector, such as the 

10 connector 700 of Figure 33, in the reception site 30', 30"or 30"' to extract the virtual objects. 

11 In this embodiment, the spread spectrum signal is transmitted in the same bandwidth as the 

12 video or audio signal, but below the noise level. 

1 3 Alternatively, several embodiments are associated with the delivery of virtual objects 

14 using a national or local radio broadcaster's signal. The signal from the national radio 

1 5 broadcaster can be delivered to the reception site 30', 30" or 30'" using the satellite system 

16 1 122, or using abroadcast affiliate 1 122. Alternatively, the radio broadcast affiliate 1 122 can 

17 originate the signal, which can be delivered to the reception site 30', 30"or 30™, terrestrially. 

18 In one embodiment, the audio signal is an analog audio signal and the virtual objects is 

1 9 embedded in the audio signal. In another embodiment, the audio signal is a digital audio signal 

20 and the virtual objects are carried as an independent data stream. In yet another embodiment, 

2 1 the virtual objects are embedded in a sub-carrier of the analog audio broadcast. In another 

22 embodiment, the signal is a spread spectrum signal containing a digital data stream, requiring 

23 an appropriate spread spectrum receiver modular connector 700 in the reception site 30', 30" 

24 or 30'" to extract the virtual objects. In this embodiment, the spread spectrum signal is 

25 transmitted in the same bandwidth as the audio signal, but below the noise level. 

26 A local insertion center 20 or multiple local insertion centers may optionally be used 

27 to insert virtual objects into content 36 provided by an operations center 1 0 or another local 

28 insertion center 20, and any other content source. A local insertion center 20 may perform the 
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1 same functions as an operations center 10. Figure 32 depicts a local insertion center 20. As 

2 shown in Figure 32, the local insertion center 20 includes a virtual object location definer 100', 

3 a virtual object selector 200', and a targeted virtual object management system 300'(TVOMS) 

4 which are identical to the virtual object location definer 100, a virtual object selector 200, and 

5 a targeted virtual object management system 300 (TVOMS) of an operations center 10. A 

6 local insertion center 20 may detect existing virtual object locations in content 36 and replace 

7 existing virtual objects with new virtual objects, delete existing virtual objects, or add new 

8 virtual objects in existing virtual object locations and target the virtual objects to reception sites 

9 or groups of reception sites. Alternatively, a local insertion center 20 may create new virtual 

10 object locations and insert and target virtual objects within these new virtual object locations 

1 1 using the processes defined for the operations center 10. 

12 Figure 33 depicts an example of a reception site 30 in more detail The modular 

13 connector 700 may handle all interactions with a reception site 30. Programming content 36 

14 with virtual object locations and metadata packets containmg placement guided and 

15 retrieval plans are received by the reception site modular connector 700 and passed to the 

1 6 virtual object extractor processor 780. The virtual object extractor processor 780 removes 

1 7 any virtual objects from the received signal and the retrieval plan information and routes the 

18 virtual objects and retrieval plan to the storage management processor 710. The storage 

1 9 management processor 710 uses the retrieval plan to determine which virtual objects are 

20 destined to the reception site 30 and saves the required virtual objects in virtual object storage 

21 720. In an alternative embodiment, virtual objects maybe received by the reception site 30 

22 independent of the programming content 36. 

23 The programming content 36 with virtual object locations is then passed to the virtual 

24 object location detector processor 750. Information received about virtual object locations is 

25 extracted from the programming content 36 andpassed to the selector processor 740 which 

26 coordinates with the storage management processor 7 1 0 to determine the appropriate virtual 

27 object 38 to place into each virtual object location 37 based on placement guidelines and 

28 available virtual objects stored in the virtual object storage 720. The storage management 
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1 processor 7 1 0 retrieves the appropriate virtual object 3 8 for one or more virtual object location 

2 37 contained in the content 36 from the virtual object storage 720. Virtual objects are passed 

3 from the storage management processor 710 to the virtual object insertion processor 

4 760. 

5 Programming content 36 with virtual object locations is passed from the virtual object 

6 location detector processor 750 to the content buffer 790 where the programming content 36 

7 is stored for a fixed period of time and then played out of the content buffer 790 to the virtual 

8 object insertion processor 760. If a virtual object 38 is available for placement in a virtual 

9 object location 37, the virtual object 38 is inserted into the appropriate virtual object location 

10 37 by the virtual object insertion processor 760. 

11 In one embodiment, the virtual object location 37 may require that an embedded 

1 2 virtual object 38 be placed within the content 36. The virtual object insertion processor 760 

1 3 may use techniques for the insertion of embedded virtual objects which are described in detail 

14 in U.S. Patents 5,953,076, to Astle, Brian; andDas, Subhodev; titled System and Method of 

1 5 Real Time Insertions into Video Using Adaptive Occlusion with a Synthetic Reference Image; 

1 6 5,892,554, to DiCicco, Darrell; and Fant, Karl; entitled System and Method for Inserting 

17 Static and Dynamic Images into alive Video Broadcast; 5,5 15,485, to Luquet, Andre; and 

1 8 Rebuffet, Michel; entitled Method and Device for Modifying a Zone in Successive Images; 

19 5,903,317, to Shark, Avi; andTamir, Michael; entitled Apparatus and Method for Detecting, 

20 Identifying and Incorporation Advertisements in a Video; and the MPEG4 standard, the 

2 1 disclosure of which are hereby incorporated by reference. 

22 In another embodiment, when the virtual object location 37 may require that an 

23 overlaid virtual object 38 be placed within the content 36. The virtual object insertion 

24 processor 760 may use techniques for the overlaying of virtual objects which are described in 

25 detailinU.S. Patents4,319,266toBannister, RichardS.;entitledChromaKeying System; 

26 4,999,709 to Yamazaki, Hiroshi; and Okazaki, Sakae; entitled Apparatus for Inserting Title 

27 Pictures; 5,249,039, to Chaplin, Daniel J.; entitled Chroma Key Method and Apparatus; and 

28 5,233,423 to Jernigan, Forest E.; and Bingham, Joseph; entitled Embedded Commercials 
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within a Television Receiver using an Integrated Electronic Billboard, the disclosure of which 
are hereby incorporated by reference. Prograrruning content 36 with embedded andoverlaid 

3 virtual objects is passed to an optional interactive object processor 770. 

4 If an external trigger is received indicating the subscriber has selected an interactive 

5 virtual object 38, the interactive object processor 770 processes the request and passes the 

6 subscriberrequestto the modular connector 700. The modular connector 700, in turn, passes 

7 the request to the interactive object servicing center 60. The interactive object servicing center 

8 60 will process the request and may respond back to the reception site 30 with an interactive 

9 response. Preferably, whenavirtualobject38isplac;edmtoavirmalobjectlocation37,the 
selector processor records the event in the placement log 730. The placement log 730 

1 1 provides viewing data to the local data collection center 40 or the central data collection center 

12 50, where the information can be used for future virtual object targeting or billing of virtual 

13 object providers, for example, advertisers. The selector processor 740 can be provided 

14 targeting algorithm updates from external sources. 

15 A lQ cal data collection center 40 is depicted in Figure 34. The local data collection 

16 center 40 collects, processes, and stores data from reception sites, from a central data 

17 collection center 50, or other sources. The data collected about reception sites may be 

1 8 provided to a local insertion center 20 to be used in targeting virtual objects in content 36. 

1 9 Alternatively, the data collected from receptions site may be provided to a central data 

20 coU ^tioncenter50tobeusedmtargetmgvirmalobjectsincontent36bya^ 

21 10. As shown in Figure 34, communications to and from the local data collection center 40 

22 overadeliverynetworkmaybedoneusingmodularconnector700. Aninterface41 receives 

23 information from reception sites. The interface 41 can include a workstation, such as the 

24 workstation 44, for example, from which an operator manually enters reception site 

25 irrformation. Alternately, reception site information can be automatically entered at the interface 

26 41 by downloading from an off-site database, the Internet, a storage medium, such as a 

27 CD-ROM or a floppy disk, and by collecting the information directly from the individual 

28 reception sites using modular connector 700. A processor 42 processes the received 
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1 reception site information and organizes the information for use and stores information in 

2 database 43. 

3 A central data collection center 50 is depicted in Figure 35. The central data 

4 collection center 50 collects, processes, and stores data from reception sites, from local data 

5 collection centers, or other sources. The data collected about reception sites may be provided 

6 to a local insertion center 20 or local data collection center 40 to be used in targeting virtual 

7 objects in content 36. Alternatively, the data collected from reception site may be provided 

8 to an operations center 10 to be used in targeting virtual objects in content 36. As shown in 

9 Figure 34, communications to and from the central data collection center 50 over a delivery 

10 networkmaybedoneusingmodularconnector700. Aninterface51 receives information 

1 1 about reception sites. The interface 5 1 can include a workstation, such as the workstation 54, 

1 2 for example, from which an operator manually enters reception site information. Alternately, 

1 3 reception site information can be automatically entered at the interface 5 1 by downloading from 

1 4 an off-site database, the Internet, a storage medium, such as a CD-ROM or a floppy disk, and 

15 by collecting the information directly from the individual reception sites using modular connector 

16 700. A processor 52 processes the received reception site information and organizes the 

17 information for use and stores information in database 53. 

1 8 An interactive object servicing center 60 is depicted in Figure 36. The interactive 

19 object servicing center 60 processes interactive requests and formulates responses to such 

20 requests. Figure 37 presents the process the interactive object servicing center 60 performs. 

21 Theprocessbeginswithblock4500. Inblock4501,theinteractiveobjectservicingcenter60 

22 receives interactive requests from reception sites. In block 4502, the interactive object 

23 servicing center 60 determines the appropriate action to be performed based on the received 

24 interactive request. In block 4503, the interactive object servicing center 60 performs the 

25 appropriate action based on the received interactive request. In block 4504, the interactive 

26 object servicing center 60 replies to the requesting reception site with an interactive response. 

27 As shown in Figure 36, communications to and from the interactive object servicing 

28 center 60 over a delivery network may be done using modular connector 700. An interface 
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1 61 receives interactive requests from reception sites. The interface 61 can include a 

2 workstation, such as the workstation 64, for example, from which an operator manually enters 

3 interactive request behavior for the interactive object servicing center 60. A processor 62 

4 processes the received interactive request, performs the appropriate action, retrieving 

5 information from database 63 to perform the actions and storing transaction information in 

6 database 63 to record the transaction event. 

7 As an alternative to the above virtual object targeting, virtual objects may be provided 

8 to reception sites 30. The virtual objects maybe stored in memory at the reception site 30 for 

9 later display during display of a video program. Virtual object placement within the video at 

1 0 the virtual object locationmay be performed atthe reception site 30 uponreceipt of the video, 

1 1 sometime after receipt of the video, or during the video program display. Virtual object 

1 2 insertion may also occur upon receipt of the video, sometime after receipt of the video, at the 

13 time of storage, or during video program display. 

14 In one embodiment, the reception site 30 maybe provided with a retrieval plan that 

1 5 is also stored in the memory. Referring to Figure 3 3, a virtual obj ect placement plan, which 

1 6 may be similar to the earlier described retrieval plan, is used by the reception site 5 s processor 

1 7 760 to determine which of the stored virtual objects is to be displayed in specific virtual object 

1 8 locations in the video program. The comparison includes analysis of virtual obj ect features and 

1 9 requirements of the specific virtual object locations in the video program. Using the placement 

20 plan, the reception site 30 may make virtual object assignments in advance of video program 

21 display. 

22 As yet another alternative, the determination of which virtual object to select for 

23 insertion into the video program may be done "on-the-fly." In this alternative, virtual object 

24 information may be provided to the reception site 30 coincident with reception of the video 

25 program, and the processor 760 at the reception site 3 0 determines a best match between the 

26 stored virtual objects and the currently supplied virtual object information. 

27 The video program with virtual obj ect locations may be displayed multiple times at the 

28 reception site 30. The virtual obj ects displayed in the virtual object locations may be made to 
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1 vary with one or more of the multiple displays of the video program. For example, the 

2 reception site 3 0 may initially display the video program with a first virtual obj ect in a specific 

3 virtual object location. Subsequent to the initial display of the video program, the reception site 

4 30 may receive updated virtual objects for insertion into the video program. A subsequent 

5 display of the video program may then result in selection of a second virtual obj ect, different 

6 from the first virtual object, in the specific virtual object location. 

7 The reception site 3 0 may store programs watched data, virtual obj ect data, and any 

8 other viewer-specific data. The processor 760 at the reception site 3 0 may then use this stored 

9 viewer-specific data to adjust the placement plan or to modify the on-the-fly virtual object 

1 0 placement. For example, if the reception site 3 0 stores more than one virtual obj ect that may 

1 1 be inserted into a virtual obj ect location, and if, the processor 760 may use the viewer-specific 

1 2 data to determine which of the virtual objects should be displayed in the virtual obj ect locations. 

1 3 The determination may be based on earlier described algorithms and algorithms described in 

14 U.S. Patent Application Serial Number 08/735,549, METHOD AND APPARATUS FOR 

1 5 TARGETING ADVERTISING, which is incorporated by reference. 

1 6 A variety of virtual obj ect targeting delivery systems have been described. One of 

17 ordinary skill in the art will recognize that the above description is that of preferred 

1 8 embodiments of the invention and the various changes and modification may be made 

1 9 thereto without departing from the spirit and scope of the invention as defined in the following 

20 claims. 
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1 What is claimed is: 

2 1 . A method for placing virtual objects in virtual object locations in a video program at 

3 a viewer's terminal, comprising: 

4 receiving one or more virtual obj ects for use with one or more of the virtual obj ect 

5 locations; 

6 storing the virtual objects in the viewer's terminals; and 

7 inserting one or more of the received virtual obj ects into one or more of the virtual 

8 object locations during a display or storage of the video program at the viewer's terminal. 

9 2. The method of claim 1 , wherein the step of inserting comprises: 

10 selecting a specific virtual object from the one or more virtual objects. 

11 3 . The method of claim 2, further comprising recording virtual obj ects watched data at 

1 2 the viewer 5 s terminal. 

13 4. The method of claim 4, further comprising adjusting the selecting step based on the 

14 recorded virtual objects watched data. 

15 5. The method of claim 1 , further comprising: 

16 receiving updated virtual objects at the viewer's terminal; and 

1 7 storing the updated virtual objects in the viewer's terminal. 

18 6. The method of claim 1 , wherein at least one virtual object is an interactive virtual 

19 object including a link to a location remote from the viewer's terminal, further 

20 comprising: 

21 receiving an activation of the interactive virtual object; and 

22 connecting the viewer's terminal to the remote location. 
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The method of claim 6, wherein the remote location is an Internet web site. 



2 8 . The method of claim 1 , wherein the viewer 5 s terminal is one of a set top terminal, a 

3 television, apersonal computer, a satellite television receiver, a wireless telephone, an electronic 

4 book reader, and a PDA device. 

5 9. A terminal that receives virtual obj ects and video programs having virtual obj ect 

6 locations and places the virtual objects into the video programs, comprising: 

7 a receiver that receives the virtual objects and the video programs; 

8 a memory that stores the virtual objects; and 

9 a processor that inserts the virtual objects into the virtual object locations during a 

1 0 display of the video programs. 

1 1 10. Theterminal of claim 9, wherein the video programs include a virtual objectplacement 

12 plan, the processor comprising a comparison module that compares the virtual object 

13 placement plan and the stored virtual objects to determine a specific virtual object for 

14 placement in a specific virtual object location. 

15 11. The terminal of claim 10, wherein the virtual obj ect placement plan is stored in the 

16 memory. 

17 12. The terminal of claim 9, wherein the receiver receives updated virtual objects and the 

1 8 memory stores the updated virtual objects. 

19 13. The terminal of claim 9, wherein the processor comprises a virtual obj ects watched 

20 module that determines virtual objects watched at the terminal, the virtual objects watched data 

2 1 stored in the memory. 
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1 14. The terminal of claim 13, wherein the processor adjusts the virtual object placement 

2 plan based on the stored virtual objects viewed data. 

3 15. The terminal of claim 9, wherein one or more virtual obj ects are interactive virtual 

4 objects, the interactive virtual objects including a link from the terminal to a remote 

5 location. 

6 16. The terminal of claim 15, wherein the remote location is an Internet web site. 

7 17. The terminal of claim 9, wherein the terminal is one of a set top terminal, a television, 

8 a personal computer, a satellite television receiver, a wireless telephone, and electronic book 

9 reader, and a PDA device. 

10 18. A method for placing virtual objects into video programs at a viewer' s terminal, 

1 1 comprising: 

12 receiving one or more virtual objects; 

13 storing the received virtual objects; 

1 4 receiving a video program including one or more virtual object locations, the video 

1 5 program including virtual obj ect information for placement of virtual objects into the video 

16 program; 

1 7 comparing the virtual obj ect information and the received virtual obj ects to select 

1 8 virtual objects for placement in the virtual object locations; and 

19 inserting the selected virtual objects into the virtual object locations. 

20 19. The method of claim 18, wherein a viewer receives virtual obj ects for display based 

2 1 on viewer information including one or more of programs watched data, virtual objects watched 

22 data, viewer demographic data, and viewer entered data. 
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1 20. The method of claim 1 8, further comprising: 

2 gathering virtual objects watched data and programs watched data; and 

3 storing the virtual objects watched data and the programs watched data in the 

4 viewer 5 s terminal. 

5 21. The method of claim 1 8, wherein the video program is stored and the inserting step 

6 occurs while the video program is stored. 

7 22. A method for placing virtual obj ects into video programs at a viewer' s terminal, 

8 comprising: 

9 receiving one or more virtual objects; 

1 0 receiving a video program including one or more virtual obj ect locations, the video 

1 1 program including virtual obj ect information for placement of virtual obj ects into the video 

12 program; 

1 3 comparing the virtual obj ect information and the received virtual obj ects to select 

14 virtual objects for placement in the virtual object locations; and 

1 5 inserting the selected virtual objects into the virtual object locations. 

16 23 . The method of claim 22, further comprising storing the received one or more virtual 

1 7 obj ects in the viewer 5 s terminal. 

18 24. The method of claim 22, wherein the video program is stored in the viewer' s terminal, 

19 and wherein the inserting step occurs during storage of the video program. 

20 25 . The method of claim 22, wherein the inserting step occurs during a display of the video 

21 program. 
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1 26 . The method of claim 22, wherein the inserting step occurs during receipt of the video 

2 program. 

3 27. The method of claim 22, wherein the video program is displayed multiple times atthe 

4 viewer' s terminal, and wherein virtual objects inserted into the video object locations vary with 

5 one or more of the multiple displays of the video program. 
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1 ABSTRACT 

2 A method and an apparatus are used to place virtual objects in video programs. 

3 Programs are selected and virtual object locations are defined in the selected programs. The 

4 available virtual objects available are categorized and the categories are correlated to 

5 subscriber information. When a frame of a program includes a virtual obj ect location, a default 

6 or an alternate virtual object is displayed. The virtual object location may change over space 

7 or time. The virtual object may be interactive, and may be used to link a subscriber to a remote 

8 location, such as an Internet web site. An operations center or a cable headend may generate 

9 a group assignment plan that assigns the subscribers ' terminals to groups, based on factors such 

10 as area of dominant influence and household income. A placement plan may instruct the 

1 1 television terminals to display the desired virtual obj ect. The television terminals may record 

1 2 which virtual obj ects were displayed, and may report this information to the cable headends 

1 3 and the operations center. The reported information is used to generate billing for commercial 

1 4 advertisers, and to analyze viewer watching habits. The invention uses upstream data reception 

1 5 hardware, databases and processing hardware and software, and corresponding features in the 

16 televisions to accomplish these functions. 
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As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name, 

I believe I am the original, first and sole inventor (if only one name is listed below) or an original, 
first and joint inventor (if plural names are listed below) of the subject matter which is claimed and for 
which a patent is sought on the invention entitled 

METHOD AND APPARATUS FOR PLACING VIRTUAL OBJECTS 

the specification of which 

(check one) 

[H is attached hereto. 

□ was filed on as United States Application No. or PCT International 

Application Number 



and was amended on 



(if applicable) 



I hereby state that I have reviewed and understand the contents of the above identified specification, 
including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose to the United States Patent and Trademark Office all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1 .56. 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 119(a)-(d) or 
Section 365(b) of any foreign application(s) for patent or inventor's certificate, or Section 365(a) of 
any PCT International application which designated at least one country other than the United States, 
listed below and have also identified below, by checking the box, any foreign application for patent or 
inventor's certificate or PCT International application having a filing date before that of the application 
on which priority is claimed. 

Prior Foreign Application (s) Priority Not Claimed 
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(Number) (Country) (Day/Month/Year Filed) 

■ □ 

(Number) (Country) (Day/Month/Year Filed) 
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(Number) (Country) (Day/Month/Year Filed) 
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- I hereby claim the benefit under 35 U.S.C. Section 119(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) 


(Filing Date) 




(Application Serial No.) 


(Filing Date) 




(Application Serial No.) 


(Filing Date) 




I hereby claim the benefit under 35 U. S. C. Section 120 of any United States application(s), or 


Section 365(c) of any PCT International application designating the United States, listed below and, 


insofar as the subject matter of each of the claims of this application is not disclosed in the prior 


United States or PCT International application in the manner provided by the first paragraph of 35 


U.S.C. Section 112, I acknowledge the duty to disclose to the United States Patent and Trademark 


Office all information known to 


me to be material to patentability as defined in Title 37, C. F. R., 


Section 1.56 which became available between the filing date of the prior application and the national 


or PCT International filing date of this application: 




09/597,893 


June 19, 2000 


Pending 


(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 


09/054,419 


April 3, 1998 


Pending 


(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 


08/735,549 


October 23, 1996 


Pending 


(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued thereon. 
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I hereby claim the benefit under 35 U.S.C. Section 119(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) 


fFilina Date} 


(Application Serial No.) 


(Filing Date) 


(Application Serial No.) 


(Filing Date) 



I hereby claim the benefit under 35 U. S. C. Section 120 of any United States application(s), or 
Section 365(c) of any PCT International application designating the United States, listed below and, 
insofar as the subject matter of each of the claims of this application is not disclosed in the prior 
United States or PCT International application in the manner provided by the first paragraph of 35 

p U.S.C. Section 112, I acknowledge the duty to disclose to the United States Patent and Trademark 
Office all information known to me to be material to patentability as defined in Title 37, C. F. R., 

yl Section 1.56 which became available between the filing date of the prior application and the national 

O or PCT International filing date of this application: 



08/160,280 


December 2 , 1993 


Issued 


(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


07/991,074 


December 9, 1992 


Pending 


(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 


(Application Serial No.) 


(Filing Date) 


(Status) 

(patented, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, 1 hereby appoint the following attorney(s) and/or 
agent(s) to prosecute this application and transact all business in the Patent and Trademark Office 
connected therewith, (list name and registration number) 



John W. Ryan, Reg. No. 33,771 

Aldo Noto, Reg. No, 35,628 

W. Robinson H. Clark, Reg. No. 41,530 

John K. Harrop, Reg. No. 41,817 

Ki S. Kim, Reg. No. 43,450 

Christopher McWhinney, Reg. No. 42,875 

Matthew Phillips, Reg. No. 43,403 

Ami P. Shah, Reg. No. 41,143 

Lance L. Vietzke, Reg. No. 36,708 

Sean S. Wooden, Reg. No. 43,997 



All of the law firm of: 
DORSEY & WHITNEY LLP 
1001 Pennsylvania Avenue, N.W. 
Suite 300 South 
Washington, D.C. 20004 



Send Correspondence to: John K - Harr °P 

DORSEY & WHITNEY LLP 

1001 Pennsylvania Avenue, N.W., Suite 300 South 

%i Washington, D.C. 20004 

O Direct Telephone Calls to: (name and telephone number) 
m John K. Harrop (202) 824-8821 



Full name of sole or first inventor 
HENDRICKS, John S. 

Sole or first inventor's signature Date 



Residence 

Potomac, Mongtomery County, Maryland U.S.A. 

Citizenship 
United States 

Post Office Address 

8723 Persimmon Tree Road 



Potomac, Maryland 20854 



Full name of second inventor, if any 
BONER, Alfred E. 

Second inventor's signature Date 



Residence 

Bethesda, Montgomery County, Maryland U.S.A. 

Citizenship 
United States 

Post Office Address 

8300 Bradley Boulevard 



Bethesda, Maryland 20817 
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Full name of third inventor, if any 
McCOSKEY, John S. 




Third inventor's signature 


Date 




Residence 

Castle Rock, Douglas County, Colorado U.S.A. 




Citizenship 
United States 




Post Office Address 
4692 N. Lariat Drive 




Castle Rock, Colorado 80104 










Full name of fourth inventor, if any 
ASMUSSEN, Michael L. 




Fourth inventor's signature 


Date 




Residence 

Oak Hill, Fairfax County, Virginia U.S.A. 


01 


Citizenship 
United States 




Post Office Address 

2627 Meadow Hall Drive 




Oak Hill, Virginia 20171 










Full name of fifth inventor, if any 




Fifth inventor's signature 


Date 


O 


Residence 




Citizenship 




Post Office Address 














Full name of sixth inventor, if any 




Sixth inventor's signature 


Date 




Residence 




Citizenship 




Post Office Address 
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